Package: laminar-py
Special thanks to Jim Fan for relinquishing the name "laminar" to me on pypi.org. Much appreciated, Jim!laminarLaminar seeks to take most of the hassle out of parallel processing in Python by providing user-friendly parallelization functionality.Module Functionsresults = laminar.iter_flow(my_function, my_iterable)results = laminar.list_flow(my_function, my_list_of_data)Class Usagemy_lam = laminar.Laminar()my_lam.add_process("process_1", function_1, my_data)my_lam.add_process("process_2", function_2, my_other_data)my_lam.launch_processes()results = my_lam.get_results()UsageInstallationLaminar is delivered as a package. To install, activate your preferred environment, then use:pip install laminarLaminar'slaminarmodule only requires one third-party library, which isnumpy. laminar_examples, a module with some practice functions and data objects, also requirespandas. Both libraries will be automatically included in the laminar installation.ImportingYou can use laminar by placingfrom laminar import laminarorimport laminar.laminar as <some_alias>at the top of your python file. In order to practice/test laminar with built-in functions and data, placefrom laminar import laminar_examplesorimport laminar.laminar_examples as <some_alias>at the top of your python file.If only using the classLaminaryou can import it directly withfrom laminar.laminar import Laminar.Using laminarlaminar currently consists of a classLaminaras well as two module functions that are designed to work with different data configurations,laminar.iter_flowandlaminar.list_flow.The Laminar class provides an instance that manages distinct processes and stores results. Class methods are available that allow the user to view, drop, clear, and launch processes.To use the Laminar class, create a Laminar instance:my_lam = Laminar()Laminar class declarations have one optional argument for number of cores, which defaults to the number of cores on the current machine. Thus, if the user only wants to utilize two cores, the declaration would be:my_lam = Laminar(2)To add a process to the object's process batch, simply use the add_process() class method, which is very similar to the module function calls listed below, except add_process() also requires the user to pass a string as the name of the process. This name can be any string.my_lam.add_process('process_1', function_1, my_data)If more processes are added than the number of cores available, the process batch acts like a first in/first out queue. The most recent process will be added and the first process added to the batch will be removed.Both of the module functions accept *args and **kwargs, which should be passed afterdata, so iffunctiontakes arg1 and arg2, like:function(arg1, arg2)you should calllaminarlike so:laminar.iter_flow(function, data, arg1, arg2)orlaminar.iter_flow(function, data, arg1=arg1, arg2=arg2)or in the case of *args with **kwargslaminar.iter_flow(function, data, arg1, arg2, kwarg=other_arg)laminar.iter_flowis designed to work with a single iterable, such as a pandas DataFrame, a python list, etc. When you pass an iterable tolaminar.iter_flow, it will automatically break your data up into chunks based on how many cores your machine has. It then queues up each chunk to be given to a core, which performs the work, then passes the data back as a descriptive dictionary of results. For example, a list of 1,000,000 integers is broken into chunks of length 250,000 on a machine with four cores. Each chunk is summed (as an example) by a core, and the results from each core are returned in a dict of size N = # cores. You are then able to combine the results in whatever way fits the computation that you need. For example, if the function passed tolaminar.iter_flowcomputes the sum, then the values in the results dict should be summed to produce a total for the entire iterable.Laminar Class DefinitionAttributeDescriptioncoresNumber of cores available in an instance. This can be set manually in the instance declaration; it defaults tocpu_count(), which is number of cores available on your machine.resultsDictionary that holds the results from thelaunch_processesmethod. Initializes to an empty dict._processescollections.OrderedDict()that holds processes added byadd_process()._queuemultiprocessing.Queue()that manages parallel processes.MethodArgument(s)ReturnsDescriptionadd_process()name: str,function: Callable,dataset: Collection,*args,**kwargs)NoneAdd a named process to an instance's process pool. Process must include a name, function, and some data (in reality, this can be anything).show_processes()NoneNoneDisplays processes currently in instance process pool.drop_process()name: strNoneRemoves process with name ofnamefrom instance process pool.clear_processes()NoneNoneRemoves all processes from instance process pool.launch_processes()Nonestr: "Processes finished."Run all instance processes in parallel.get_results()Noneself.results: dictReturns the instance results dictionary.clear_results()NoneNoneRemoves all results from instance results dictionary.Module Function ExamplesTo illustrate how one would use laminar in their workflow, we'll use some premade functions and data structures located inlaminar_examples. To shorten the following code examples up, we'll importlaminar_examplesas an aliasleand use this alias throughout the rest of this readme.from laminar import laminar_examples as lelaminar_examples.single_totalle.single_totalis a simple function that accepts a single iterable and returns the sum total of the values in that iterable.le.single_total([1, 2, 1])returns4.laminar_examples.multi_tallyle.multi_tallyis a simple funtion that accepts a Pandas DataFrame and returns the number of rows that sum to greater than 25.le.multi_tally(pd.DataFrame({'Col1': [12, 12], 'Col2': [12, 14]})returns1because the row at index 1 sums to12 + 14 = 26, which meets the function's criteria, but the row at index 0 sums to12 + 12 = 24, which does not.laminar_examples.laminar_dfle.laminar_dfis a Pandas DataFrame that constitutes 3 columns ['Col1', 'Col2', 'Col3'], each of which contains different integer values.Col1Col2Col3161127123813491451015212224616.........Example 1: Single iterable, single_total()laminar.iter_flow(le.single_total, le.laminar_df['Col1'])returns{'data[0-5]': 17,'data[12-17]': 60,'data[18-23]': 86,'data[24-29]': 115,'data[30-34]': 105,'data[35-39]': 120,'data[40-44]': 135,'data[6-11]': 37,}which is a dictionary describing the results for each section of your data. Each key/value pair in the returned dict corresponds to a segment of the iterable that was broken out and given to a process, with the key containing which portion of the data the result matches to. To complete your analysis, you can use whichever function coincides with the intended behavior of your analysis. In this case, since we are summing values, we can usesum().The end result can look like one of these examples, although it doesn't have to:result = sum(laminar.iter_flow(le.single_total, le.laminar_df['Col1']).values())orresult = laminar.iter_flow(le.single_total, le.laminar_df['Col1'])result = sum(result.values())whereresult = 675Example 2: Pandas DataFrame, multi_tally()laminar.iter_flow(le.multi_tally, le.laminar_df)returns{'data[0-5]': 3,'data[12-17]': 6,'data[18-23]': 6,'data[24-29]': 6,'data[30-34]': 5,'data[35-39]': 5,'data[40-44]': 5,'data[6-11]': 6,}which is a dict of counts. Each count is the return value for a segment of the data that was broken out and given to a process. To complete your analysis, you can use whichever function coincides with the intended behavior of your analysis. In this case, since we are counting values, it makes sense to usesum().The end result can look like one of these examples, although it doesn't have to:result = sum(laminar.iter_flow(le.multi_tally, le.laminar_df).values())orresult = laminar.iter_flow(le.multi_tally, le.laminar_df)result = sum(result.values())whereresult = 42Example 3: List of single iterables, single_total()laminar.list_flow(le.single_total, [le.laminar_df[col] for col in le.laminar_df.columns])returns{'data_position_0': 675,'data_position_1': 1800,'data_position_2': 2925,}which is a list of the totals for each column inle.laminar_df. With this usage, a user can pass a list of iterables tolist_flow; each iterable will be passed to its own process. This is useful for when a user intends to use the same function on multiple iterables, which can be columns in the same DataFrame, or independent lists.laminar.list_flow(laminar_examples.single_total, [laminar_examples.laminar_df[col] for col in laminar_examples.laminar_df.columns])returns[675, 1800, 2925], which is a list of the totals for each column. With this usage, a user can pass a list of iterables tolist_flow; each iterable will be passed to its own process. This is useful for when a user intends to use the same function on multiple iterables, which can be columns in the same DataFrame, or independent lists.columns_list = [le.laminar_df[col] for col in le.laminar_df.columns]result = laminar.list_flow(le.single_total, columns_list)whereresult = {'data_position_0': 675, 'data_position_1': 1800, 'data_position_2': 2925}Example 4: List of Pandas DataFrames, multi_tally()laminar.list_flow(le.multi_tally, [le.laminar_df]*3)returns{'data_position_0': 42,'data_position_1': 42,'data_position_2': 42,}.The result values are the same because we passed a list of 3 identical DataFrames; feel free to test this with different DataFrames of your own making.data_frames_list = [le.laminar_df]*3result = laminar.list_flow(le.multi_tally, data_frames_list)whereresult = {'data_position_0': 42, 'data_position_1': 42, 'data_position_2': 42}BenchmarksTo date, laminar has been tested against traditional iterative analysis on the following functions:String search function: count_snps()ParametersFiles:sample-1_S1_R1_001.fastq.gzsample-1_S1_R2_001.fastq.gzTotal size of files:26MLength of Pandas DataFrame (going forward referred to as pd.DataFrame) object representation of combined files:224706 rowsResults:Traditional count_snps(pd.DataFrame): 42.6 secondsParallelized laminar.iter_flow(count_snps, pd.DataFrame): 17.49 secondsPercent speedup: 58.96% fasterFinal NotesWhich laminar tool a user will use depends on the structure of their data and the function that will be applied to that data.laminar.list_flowis not confined to operating on Pandas DataFrames; any list of iterable data objects can be passed to list_flow.A basic rule of thumb is to uselaminar.iter_flowfor a single data object that one wishes to break into pieces in order to process it faster.laminar.list_flowis to be used in a situation where the user has multiple data objects that he or she wishes to be analyzed by the same function in parallel.