Package: sample-cli
sampleis a command-line tool for sampling data from a large,
newline-separated dataset (typically a CSV-like file).Installationsampleis distributed withpip. Once you’ve installedpip,
simply run:> pip install sample-cliand sample will be installed into your Python environment.Usagesamplerequires one argument, the input file. If the input file
is-, data will be read from standard input (in this case, only
the reservoir and approximate algorithms can be used).Simple ExampleTo take a sample of size 1000 from the filebig_data.csv,
runsampleas follows:> sample -n 1000 big_data.csvThis will print 1000 random lines from the file to the terminal.File RedirectionUsually we want to save the sample to another file instead.sampledoesn’t have file output built-in; instead it relies
on the output redirection features of your terminal. To save
tobig_data_sample.csv, run the following command:> sample -n 1000 big_data.csv > big_data_sample.csvHeader RowsCSV files often have a header row with the column names. You can pass
the-rflag tosampleto preserve the header row:> sample -n 1000 big_data.csv -r > big_data_sample.csvRarely, you may need to sample from a file with a header spanning
multiple rows. The-rargument takes an optional number of
rows to preserve as a header:> sample -n 1000 -r 3 data_with_header.csv > sample_with_header.csvNote that if the-rargument is directly before the input filename,
it must have an argument or else it will try to interpret the input
filename as the number of header rows and fail. Putting the-rargument
after the input filename will avoid this.Random SeedThe output ofsampleis random and depend on the computer’s random
state. Sometimes you may want to take a sample in a way that can be
reproduced. You can pass a random seed tosamplewith the-sflag
to accomplish this:> sample -s 45906345 data_file.csv > reproducable_sample.csvSampling AlgorithmsAlgorithm Comparisonsampleimplements three sampling algorithms, each with their own strengths
and weaknesses.ReservoirApproximateTwo-passFlag--res--app--tpstdin-compatibleyesyesnospace complexityO(ss * rs)O(1)O(ss)fixed sample sizecompatiblenot compatiblecompatiblefractional sample sizenot compatiblecompatiblecompatibleFor space complexity,ssis the number of records in the sample andrsis the maximum size of a record.Reservoir SamplingReservoir sampling (Random Sampling with a Reservoir (Vitter 85))
is a method of sampling from a stream of unknown size where the sample size is
fixed in advance. It is a one-pass algorithm and uses space proportional to the
amount of data in the sample.Reservoir sampling is the default algorithm used bysample. For consistency,
it can also be invoked with the argument--reservoir.If reservoir sampling, the sample size must be fixed rather than fractional.Example:> sample --reservoir -n 1000 big_data.csv > sample_data.csvApproximate SamplingApproximate sampling simply includes each row in the sample with a probability
given as the sample proportion. It is a stateless algorithm with minimal space
requirements. Samples will have on average a size offraction * population_size,
but it will vary between each invocation. Because of this, approximate sampling
is only useful when the sample size does not have to be exact (hence the name).Example:> sample --approximate -f 0.15 my_data.csv > my_sample.csvEquivalently, supply a percentage instead of a fraction by switching the-fto a-p:> sample --approximate -p 15 my_data.csv > my_sample.csvTwo-Pass SamplingTwo-pass sampling is allowed two passes, first to count the number of records
(ie. the population size) and second to emit the records which are part of the
sample. Because of this it is not compatible withstdinas an input.As two-pass sampling knows the population size, it will accept the sample size
as either a fraction or a fixed number of elements.Example:> sample --two-pass -p 15 my_data.csv > my_sample.csvTwo-pass sampling uses memory proportional to the number of elements in the sample.