Package: dastr
dastrMotivationOpen-source datasets often aren't organized according to a standard directory structure. This means that, if your analysis code expects a particular format, either you have to create a new version of it that loops through files differently or you have to manually move and rename the files to match your code's expected format.dastrautomates the latter approach.How to usedastrcan be installed through pip (pip install dastr).There are 3 steps to convert between directory structures: (1) reading, (2) translating, and (3) writing.1. ReadingThere will generally be metadata encoded in a dataset's folder and file names. For example, maybe each participant's data are stored in a separate folder and these are called "p01", "p02", etc. Maybe it's a longitudinal EEG study and within each participant-specific folder are files called "pre.edf" and "post.edf".dastruses regular expression captures (with Python'srelibrary) to read this data into a table (alistofdicts) where each row (element) is a file and each column (key) is an attribute of that file (for example, participant ID and session).To read in the data from the above example, you would do the following:importdastrdata_path='/path/to/data/'files=dastr.read(path=data_path,params=[("p(.+)","participant ID"),("(.+)\.edf","session")],disp=True)The function will start from the folder specified by thepathargument and go down from there. The action to take at each level of the directory structure is specified by each elements of theparamsargument (a list of tuples, the elements of which will be strings). The first element of each tuple will be a regular expression. If there are any captures in the regular expression, they will be recorded as the attributes specified by the remaining elements of the tuple. For example:("(.+)\.edf", "session")in the above means that, once the program gets to the file/path/to/data/p01/pre.edfit should run something equivalent in this case to:curr_file["session"]=re.findall("(.+)\.edf","pre.edf")[0]If the current file/folder doesn't match the regular expression, this file is skipped. If the tuple is empty, the regular expression defaults to the wildcard character (.).dastr.read()returns alistofdictss with keysattrsandpath.pathspecifies the path to the actual file whileattrscontains another dictionary that stores the attributes. I.e., in the above example,files[0]["path"]would be"/path/to/data/p01/pre.edf"whilefiles[0]["attrs"]would be{"participant ID": "01", "session": "pre"}. To print this information as it is being read, set the optional argumentdisp=Trueas in the above (or omit this argument to print nothing).In some cases, this might be all you need. You could usedastr.flatten()to get a newlistofdicts with key-value pairs copied fromfiles[:]["attrs"], plus an additional key-value pair specifying the path of the file. This makes perfect input forDictWriterfrom Python'scsvlibrary. The resulting csv table could be read in by your analysis code, point it to each data file, and take its outputs as new columns. Or maybe you don't want to change your analysis code.2. TranslatingSuppose instead of "pre" and "post", you want to call the sessions "01" and "02". The function to use then isdastr.translate():translation={"session":{"pre":"01","post":"02"}}translated=dastr.translate(files=files,translation=translation)original=dastr.translate(files=translated,translation=translation,direction="reverse")# Or equivalently "backward", or actually anything other than "forward"3. WritingActually moving the files is very similar todastr.read()ing them. The difference is that, instead of reading attributes in usingrecaptures, you're writing them out using string formatting. For examplenew_path="/new/data/path/"destinations=dastr.write(files=files,path=new_pathparams=["alldata",("sub-%s","participant ID"),("sub-%s_ses-%s.edf","participant ID","session")])creates the variabledestinations, alistofstrings specifying the new locations of the files (though they haven't been deleted from their old locations):/new/data/path/alldata/sub-01/sub-01_ses-01.edf
/new/data/path/alldata/sub-01/sub-01_ses-02.edf
/new/data/path/alldata/sub-02/sub-02_ses-01.edf
/new/data/path/alldata/sub-02/sub-02_ses-02.edfIf you did want to delete the old files, you could add the optional argumentkey="x"to use Python'sshutil.move()(by default,keyis"c"which uses Python'sshutil.copy()). You can also setkeyto any function that takesold_path, new_pathas its arguments, or you can set it to"n"which doesn't touch the files at all (in which case you'd probably also want to setdisp=True).How to use with JSONTo avoid hard-coding the parameters and translations, you can instead specify them in .json files (thanks toGabi Hermanfor the suggestion):Reading and writing with JSONInstead of runningdastr.[read/write](...,params=params,...)you would rundastr.[read/write](...,params=dastr.json_to_params("path/to/file.json"),...)Wherefile.jsonis formatted in one of 3 ways:alistofdicts[{"pattern":"p(.+)","attrs":"participant ID"},{"pattern":"(.+)\.edf","attrs":"session"}]adictoflists{"patterns":["p(.+)","(.+)\.edf"],"attrs":["participant ID","session"]}alistoflists andstrings["alldata",["sub-%s","participant ID"],["sub-%s_ses-%s.edf","participant ID","session"]]Translating with JSONHere all you need to do is copy and paste your hard-codedtranslationvariable into a.jsonfile (replacing single quotes with doubles), and pass the location of this file todastr.translate:translated=dastr.translate(...,translation="path/to/translation.json")Runningdastr.json_to_paramswouldn't work here.