Package: ekpmeasure
Welcome to ekpmeasureA repository of analysis and computer control code for various experiments. Image above is an example of datacollectedandanalyzedusing this package.OverviewUser GuideInstallationDevelopmentImportant LinksChange LogSupportCite this codeOverviewekpmeasure is a set of control and analysis code designed to help streamline experiments. The basic idea is that in experimental work we often take data from many different sources, store it in different places, have varying degrees meta data associated with the data (even for a single type of data) and somehow(!) we are supposed to make sense of it all. We like to compare across trials, days, experimental conditions, etc. and it is very difficult to keep track of what data is where, and quickly access it when we need it. Often I find that folks end up copying and pasting raw data between excel spreadsheets and if you're not careful you will quickly lose track of which data came from where. This package's goal is to make this all easier.You may not find the experimental control code as helpful as it is relatively specific to my research in condensed matter physics (though electrical engineers or similar may find it very useful) but the analysis code is for everyone.At the heart of the analysis is theDatasetclass which is a means of manipulatingmeta data alonein order to locate which actual data you want to analyze.Datasetsdon't care about what the real data looks like, and they keep track of where different data is stored so it is easy to select which data you want to look at - only then do you retrieve the data. The real data is returned in aDataclass which allows you to group by parameters, perform calculations and much more.I am always improving this repository and if you have suggestions, I appreciate any feedback and or issues (https://github.com/eparsonnet93/ekpmeasure/issues)User GuideHere, we will walk through a brief example of how the analysis power ofekpmeasure. Scientists often have large amounts of data, stored in different locations and with varying degrees of available meta data. ekpmeasure makes it easier to access all of this data. Once created, you can easily load aDatasetSay we have some data stored in./path/to/data/which is a bunch of.csvfiles. We can open one of them (calledfilename.csv) up usingpandas.read_csvand see what's inside:importpandasaspdimportnumpyasnppath='./path/to/data/'filename='filename.csv'>>>df=pd.read_csv(path+filename)>>>df.head()This data file has some meta data associated with it. Let's assume for this example thatfilename.csvcorresponds to the following meta data:{'voltage_applied':'10.0V','pulsewidth':'100ms'}i.e.the file we have here is for a specific type of stimulus, namely an applied voltage of 10V and a pulsewidth of 100ms (doesn't matter what that actually means, it could be any type of meta data). Let's plot this specific data file. Here I want to plotPolarization(uC/cm2)_xandPolarization(uC/cm2)_yvsDRIVEVoltage_x.df.plot(x='DRIVEVoltage_x',y=['Polarization(uC/cm2)_x','Polarization(uC/cm2)_y'])Clearly there is a difference between'Polarization(uC/cm2)_x'and'Polarization(uC/cm2)_y', and for this particular case, we are interested inquantifying that differencebetween them. An easy way to do this is by looking at each of their values around'DRIVEVoltage_x = 0', and finding the difference. We will do this in two steps, first find the average of each around zerodeffind_avg_P_loop1_loop2(dataframe,center_voltage=0,window=.1):"""Find the average of `Polarization(uC/cm2)_x` and `Polarization(uC/cm2)_y` for data corresponding to `center_voltage - window <= DRIVEVoltage_x <= center_voltage + window`args:dataframe (pandas.DataFrame): DataFrame with columns `Polarization(uC/cm2)_x`, `Polarization(uC/cm2)_y` and `DRIVEVoltage_x`center_voltage (float): Center voltage to find average aroundwindow (float): Window around center voltage to averagereturns:(pandas.DataFrame): Average around center_voltage for loop1 and loop2. Columns: 'p1' and 'p2'"""p1,p2=dataframe['Polarization(uC/cm2)_x'],dataframe['Polarization(uC/cm2)_y']drivevoltage=dataframe['DRIVEVoltage_x']#find indices of where the window (center_voltage \pm window) is:indexer=(drivevoltage<=center_voltage+window)&(drivevoltage>=center_voltage-window)out={'p1':[np.mean(p1[indexer])],'p2':[np.mean(p2[indexer])]}returnpd.DataFrame(out)Next we need a function that calculates the difference.defdifference(dataframe):"""Calculate the absolute value of the difference between dataframe columns `p1` and `p2`.args:dataframe (pandas.DataFrame): DataFrame with columns `p1` and `p2`returns:(pandas.DataFrame): Difference between p1 and p2. Columns: `difference`"""returnpd.DataFrame({'difference':abs(dataframe.p1-dataframe.p2).values})Now we are ready to return to our data and plot ourimportant metriccalled difference. Doing it for the one file is easy:>>>avgdf=find_avg_P_loop1_loop2(df)>>>dif=difference(avgdf)>>>print(dif)>difference01.431387Great! Now this was forone datafilecorresponding to an applied voltage of 10V and a pulsewidth of 100ms. What if I want to this for all of my data which corresponds to a pulsewidth of 100ms, but for all values of applied voltages? Now I need to track down the filenames associated with each of these types of data, sort them by applied voltage and then do the calculations I just did - then return that all in some sort of managable object. The problem is even worse if the meta data is storedin the fileso you need to open each file in order to see what type it is.Andwhat if you have data in multiple folders all over your computer - it gets very hard to manage all this. This is whereekpmeasurecomes in. Let's assume you have already generated the meta data (this can be done, by usingekpmeasureto generate the data seeexperiment Class, or by generating if from existing data seegenerate_meta_data) and stored it in./path/to/data. We can load all of the associated meta data easily:fromekpmeasureimportanalysis>>>dset=analysis.load_Dataset('./path/to/data/')>>>dsetdsetholds all of the meta data. If we want to look at our same file as the example above, we can access it by querying our dataset: ('pump_amp'is what we called'voltage_applied'and'pump_pw'is what we called'pulsewidth)>>>dset.query('pump_amp == "10.0V" and pump_pw == "100ms"')This makes it easy to get find data based on the meta data. For example, let's find all of the data associated withjustthe pulsewidth'pump_pw'of 100ms.>>>dset.query('pump_pw == "100ms"')You can see that for this one pulsewidth, we havemanyapplied voltages ('pump_amp'). We will return to this in a moment. Let's stick with our query'pump_amp == "10.0V" and pump_pw == "100ms"'and see how we can retrieve the real data (seeget_data). Recall from above that the raw data has three important columns'Polarization(uC/cm2)_x','Polarization(uC/cm2)_y',  and'DRIVEVoltage_x'>>>tmpdset=dset.query('pump_amp == "10.0V" and pump_pw == "100ms"')>>>data=tmpdset.get_data()#retrieve the data#plot the data>>>data.plot(x='DRIVEVoltage_x',y=['Polarization(uC/cm2)_x','Polarization(uC/cm2)_y'])Same as before! But now, we canplot all of the datacorresponding to a pulsewidth of 100ms, with just a very simple modification.>>>tmpdset=dset.query('pump_pw == "100ms"')>>>data=tmpdset.get_data()#retrieve the data#plot the data>>>data.plot(x='DRIVEVoltage_x',y=['Polarization(uC/cm2)_x','Polarization(uC/cm2)_y'])and done, this is all of the data for 100ms where the color corresponds to the applied voltage ('pump_amp', ranging from 5V to 10V). No more sifting through csv's to find what you're looking for! Let's keep going. Remember, we are interested in the difference between averages around zero voltage for each datafile. We can create a workflow for this usingData.apply. To understand how this works, let's take a deeper look at what ourdataobject really is:>>>type(data)>ekpmeasure.analysis.core.DataThisekpmeasure.analysis.core.Datais integer indexed and holds the raw data as well as a definition (describing the meta data) for each index. Let's take a look:>>>tmpdset=dset.query('pump_amp == "10.0V" and pump_pw == "100ms"')>>>data=tmpdset.get_data()#retrieve the data>>>data>{0:{'definition':{'pump_amp':{'10.0V'},'pump_pw':{'100ms'},'preset_amp':{'9V'},'preset_pw':{'1ms'},'trial':{0}},'data':{'Time(ms)_x':array([0.000e+00,2.000e-03,3.000e-03,...,1.999e+00,2.000e+00,2.001e+00]),'DRIVEVoltage_x':array([-0.002747,0.013809,0.027161,...,-0.029373,-0.01564,-0.00145]),'Polarization(uC/cm2)_x':array([-0.41523,-0.410556,-0.41523,...,-0.840587,-0.779822,-0.786054]),'Time(ms)_y':array([0.000e+00,2.000e-03,3.000e-03,...,1.999e+00,2.000e+00,2.001e+00]),'DRIVEVoltage_y':array([-0.002289,0.01358,0.027771,...,-0.028152,-0.014114,-0.000992]),'Polarization(uC/cm2)_y':array([1.130391,1.138181,1.13974,...,0.622455,0.676988,0.689453])}}}We have the index (0), for which we have the 'definition' and the 'data'. 'Definition' describes the meta data associated with that index (here it matches our query, of course) and 'data' is a Dict which holds the raw data. Let's now take a look at how to calculate the difference, as before. We will do this usingData.apply.Data.applytakes a function and passes that function the 'data' key for each index in aekpmeasure.analysis.core.Dataobject. For example, in our current case, sincedatahas only one index (0) it will pass the raw data. To make this clear, I will define a function that takes a data_dict as argument, then send use it indata.applydefprint_data(data_dict):"""Print the data that we are passed and return it"""print(data_dict)returndata_dict>>>_=data.apply(print_data)>{'Time(ms)_x':array([0.000e+00,2.000e-03,3.000e-03,...,1.999e+00,2.000e+00,2.001e+00]),'DRIVEVoltage_x':array([-0.002747,0.013809,0.027161,...,-0.029373,-0.01564,-0.00145]),'Polarization(uC/cm2)_x':array([-0.41523,-0.410556,-0.41523,...,-0.840587,-0.779822,-0.786054]),'Time(ms)_y':array([0.000e+00,2.000e-03,3.000e-03,...,1.999e+00,2.000e+00,2.001e+00]),'DRIVEVoltage_y':array([-0.002289,0.01358,0.027771,...,-0.028152,-0.014114,-0.000992]),'Polarization(uC/cm2)_y':array([1.130391,1.138181,1.13974,...,0.622455,0.676988,0.689453])}this is precisely the data stored in the 'data' key on index 0 above. Now let's do something more interesting. I will use our functionfind_avg_P_loop1_loop2from before with a few key changes to make it ready for usage inData.apply. You'll notice now, I am operating onDictnotpandas.DataFrame, but otherwise everything is the same.deffind_avg_P_loop1_loop2(data_dict,center_voltage=0,window=.1):"""Find the average of `Polarization(uC/cm2)_x` and `Polarization(uC/cm2)_y` for data corresponding to `center_voltage - window <= DRIVEVoltage_x <= center_voltage + window`args:data_dict (Dict): Dict with keys `Polarization(uC/cm2)_x`, `Polarization(uC/cm2)_y` and `DRIVEVoltage_x`center_voltage (float): Center voltage to find average aroundwindow (float): Window around center voltage to averagereturns:(Dict): Average around center_voltage for loop1 and loop2. Keys: 'p1' and 'p2'"""p1,p2=data_dict['Polarization(uC/cm2)_x'],data_dict['Polarization(uC/cm2)_y']drivevoltage=data_dict['DRIVEVoltage_x']#find indices of where the window (center_voltage \pm window) is:indexer=(drivevoltage<=center_voltage+window)&(drivevoltage>=center_voltage-window)out={'p1':np.array([np.mean(p1[indexer])]),'p2':np.array([np.mean(p2[indexer])])}returnoutlet's try it out on our data.>>>data.apply(find_avg_P_loop1_loop2)>{0:{'definition':{'pump_amp':{'10.0V'},'pump_pw':{'100ms'},'preset_amp':{'9V'},'preset_pw':{'1ms'},'trial':{0}},'data':{'p1':array([-0.62944063]),'p2':array([0.80194658])}}}Exactly the same as before. Let's finish this out and get our difference. We will need to redefine ourdifferencefunction as well to operate on dicts.defdifference(data_dict):"""Calculate the absolute value of the difference between dataframe columns `p1` and `p2`.args:dataframe (Dict): Dict with keys `p1` and `p2`returns:(Dict): Difference between p1 and p2. Keys: `difference`"""return{'difference':abs(data_dict['p1']-data_dict['p2'])}and run it:>>>data.apply(find_avg_P_loop1_loop2).apply(difference)>{0:{'definition':{'pump_amp':{'10.0V'},'pump_pw':{'100ms'},'preset_amp':{'9V'},'preset_pw':{'1ms'},'trial':{0}},'data':{'difference':array([1.43138722])}}}which matches our result from before. Now, the fun part. Let's do it on all of the data corresponding to a pulsewidth of 100ms for any applied voltage. All we have to do is define our definition of thedataobject.>>>tmpdset=dset.query('pump_pw == "100ms"')>>>data=tmpdset.get_data()#retrieve the data>>>diff_data=data.apply(find_avg_P_loop1_loop2).apply(difference)Now,diff_dataholds all of our differences for all of the data corresponding to a pulsewidth of 100ms. Equally importantly it is labeled by its complete meta data. Let's say we now want to plot our calculated difference (areal datavariable) vs pump_amp (ameta datavariable). This can be done byanalysis.get_vals_by_definition(Remember we can just use.plot()to plot real data vs real data)vbd=analysis.get_vals_by_definition(diff_data,definition_key='pump_amp',data_key='difference')vbdis a Dict with (key, value) pairing (voltage applied, calculated difference for that datafile). To parse such a dict, and turn it into plottable arrays, we can useanalysis.vals_by_definition_to_2darrayimportmatplotlib.pyplotasplt# we need to tell it how to convert the string for pump_amp (like "10.0V") into a float, by using the kwarg 'converter'>>>X,Y=analysis.vals_by_definition_to_2darray(vbd,converter=lambdax:float(x.replace('V','')))>>>plt.scatter(X,Y,color='blue')And there you have it, we now analyzed all of the data corresponding to 100ms! As a final example in this introduction, let's now plot all of the data. We want difference vs applied voltage colored by pulsewidth. The following code will create it:frommatplotlibimportcmfromekpmeasure.universalimportget_number_and_suffix,time_suffix_to_scientic_str# sort the pulsewidths in the datasetdefsorter(pw):number,suffix=get_number_and_suffix(pw)returnfloat(str(number)+time_suffix_to_scientic_str(suffix))# group, average over trials, sort by 'pump_pw'data=dset.get_data(groupby='pump_pw').mean().sort(by='pump_pw',key=sorter)# generate a color map for each pulsewidthcmap=cm.plasmacolors=[cmap(x)forxinnp.linspace(0,1,len(pws))]# set up a plotfig,ax=plt.subplots()fortmpdata,colorinzip(data,colors):# calculatediff_data=tmpdata.apply(find_avg_P_loop1_loop2).apply(difference)# want to plot versus 'pump_amp', a definition keyvbd=analysis.get_vals_by_definition(diff_data,definition_key='pump_amp',data_key='difference')X,Y=analysis.vals_by_definition_to_2darray(vbd,converter=lambdax:float(x.replace('V','')))ax.scatter(X,Y,color=color)Hopefully this has given you a brief introduction to how to use EKPMEASURE. Happy analyzing!For more seehere.Installation:Find the latest buildhere.pipinstallekpmeasureYou can also access.whlor.tar.gzfiles in thedist/directory directly for installation.For installation issues, please seeIssue TrackerThere are also specificexperimentsthat can be installed following installation ofekpmeasure. Please see the accompanying readme (e.g.,here) for such cases.DevelopmentWe welcome new contributors of all experience levels. Please reach out directly (e.parsonnet@berkeley.edu) to inqure about getting involved.Important LinksOfficial source code repo:https://github.com/eparsonnet93/ekpmeasureIssue tracker:https://github.com/eparsonnet93/ekpmeasure/issuesChange logVersion 0.1.6**1/23/22addedData.to_DataFrame()which allows one to convertDatatopandas.DataFrame. Each index ofDatawill be a single row in the resulting DataFrame.Version 0.1.51/12/22merge_Dataswas replaced. The older version is nowconcat_Datasas it was really just concatenation, not merging. Merging now has real meaning, to merge a set of similar data objects on a specified definition key. Please report errors as they arise.similarymerge_Datasetsis deprecated. One must useconcat_Datasets.Version 0.1.412/2/21Added experiments module withferroelectricexperiments. This contains both relaxation (_relaxation) and switching (_switching) experiments. These consist of self contained jupyter notebooks that can be installed bypython -m ekpmeasure.experiments.ferroelectric.<experiment_name>. For more see the experiment specific README forrelaxationorswitchingBug fix onData.sort11/16/21Data.applynow allows for dropping data. This can be executed by having the function in.applyreturn'None'.data_array_builder.buildnow allows one to fix lengths on 1D data by appending nans to make all data arrays the same length.analysis.plotting.add_legend_elementnow allows kwarg fontsize11/19/21Datacan now be saved. Usedata.to_ekpdat.Can be loaded asanalysis.read_ekpdatVersion 0.1.311/11/21Minor fixes for deprecations incontrol.miscVersion 0.1.211/7/21Speed improvements toDataset.remove_nonexistent_filesDataobjects can now be sorted by a definition parameterVersion 0.1.111/2/21Updates to plotting during experimental control. Now one can simply override thecontrol.experimentmethod_plotto define how plotting will take place. Here is a brief example of such an override:fromekpmeasure.controlimportplottingfromekpmeasure.controlimportexperimentimportmatplotlib.pyplotaspltclassexp(experiment):...def_plot(self,data,scan_params):ifhasattr(self,'fig')andhasattr(self,'ax'):passelse:fig,ax=plt.subplots()self.fig=figself.ax=axself.ax.scatter(scan_params['frequency'],np.mean(data['R']),color='blue')plt.show(self.fig)plotting.update_plot(self.fig)...Version 0.1.010/24/21Dataset class is no longer subclass ofpandas.DataFrame. This is to limit usage of unsupported functions.Data class updates including sorting and collapsing.One can access data or definition directly as an attribute nowData.definition, for example. One can also access pieces of information such as the real data corresponding top1asData.p1or definition keys,e.g.high_voltageasData.high_voltageExperiment class now saves a backup.csvmeta data file in addition to the pickle file in order to help with errors related to different pandas versions on various machines.SupportCode related issues (e.g. bugs, feature requests) can be created in theissue trackerMaintainer: Eric ParsonnetCitationPlease cite this work as:ekpmeasure
EricParsonnet,2022https://github.com/eparsonnet93/ekpmeasure