Package: refyre
refyre: Large scale file management energized by AIRefyre is an AI-fused Python package that provides two high level features:Easy large scale filesystem manipulationsEfficient, code-less directory structuring and restructuringEnhance your favorite Python packages such as Pandas, NumPy, Spark, and other data manipulation tools to quickly structure scattered data.FeaturesFilesystem agnostic data handshakesKickstart loading entire repositories & setting up virtual environments in a single command, your wayPerform mass operations on files such as copying, moving, zipping, POST-ing, in 1 line of codeHomebrew structured data such as Pandas DataFrames, and image datasets in a snap of your fingers (< 30 lines)Refactor, organize, and analyze periodic research experiments with zero lines of codeQuickstartSimply provide refyre with an "input specification", telling it what directories to focus onsample_input_spec.txt'''Suppose you have a directory structurea/a1.txta2.txt...b/c/c1.txtc2.txtd.txtd2.txt...You seek to analyze the a files and the c files'''[dir="a"|name="a_var"][dir="b"][dir="c"|pattern="g!c?.txt"|name="c_var"]#Glob patterns start with 'g!', regex with 'r!', no need for just normal pattern matchingHave refyre analyze the directory with the following:#Main analysis lineref=Refyre(input_specs=['sample_input_spec.txt'])#Now, have a bit of fun!a_var=ref["a_var"]c_var=ref["c_var"]print(len(a_var))#Number of files#Move all the files to another directory, copy works the same waya_var=a_var.move('dir2')#.copy() ...#Get all the files in a List[Pathlib.Path] objectsall_a_var=a_var.vals()#Automatically zip a copy of all the fileszipped_c_var=c_var.zip()print(len(zipped_c_var))#1, the zipped c_var files#Get all the parents dirsc_var_parent_dirs=c_var.dirs()print(type(c_var))#refyre.cluster.FileCluster (this is what each variable type is)#Do mass file management operations such as delete(), filter()all_a_var_and_c_vars=FileCluster(values=[])#Values are strings of filepaths you want to do operations onall_a_var_and_c_vars=a_var+c_varfiltered_c=all_a_var_and_c_vars.filter(lambdap:p.name.startswith('c'))#Delete all filesfiltered_c.delete()#Automatically account for any modifications by variablesprint(len(all_a_var_and_c_vars))And finally, after any analysis, you can use the variables to generate
specsLet's say you want to generate directories & data in the format specified byoutput_spec.txt:'''Sample output spec, createsdirectories d & e, and ports the datafrom a_var and c_var into it.'''[dir="d"|name="a_var"][dir="e"|name="c_var"]One line.ref.create_spec('output_spec.txt')Alternatively, this entire process (minus the in-between analysis) can be done through our CLI.refyre-iinput.txt-ooutput.txtMicrodocsLet's provide a quick overview of the various capabilities refyre packs.Spec Attributes:Specs, as shown above are a Pythonic way for you to feed information to refyre. Each [] represents acluster, which usually has a dir attribute specified. Attributes are seperated using the '|' seperator.As shown above, Pythonic comments can be used in a similar fashion to Python. Back to the various attributes:dir: Specifies the directory the cluster is targeting.Usually, the clusters are relative paths.You can specify the three pattern types to target multiple directoriespattern: Allows you to target specific files by specifying a template pattern. Currently, glob, regex, and "generator expressions" are supported.For glob patterns, add a 'g!' before the pattern; ex:g!*.txtFor regex patterns, add an 'r!' before the pattern ex:r!.txtGenerator expressions a simplified pattern matching, that's more humanly controllableJust one template matching -->$matches to a numberrefyre supports generator expressions the most out of the threename: The workhorse of specs. Arguably themeatof the spec. Assigns all the values to a variable specified byname.Only single variable / cluster are supportedYou can achieve 'appending' by specifying a+before the nameflags: A grab bag of various tricks you can use. You can specify as many as you want, and they work together to bring out cool cluster behaviours*mmakes a directory if it doesn't exist in a read spec*d(only during generation) deletes everything in thecurrentdirectory except for the clusters specified*dadeletes everything in thecurrent & all subclustersexcept for clusters listed*fgets all the files listed in the current directory*dgets all the directories listed in the current directory*rallows*fand*dto behave recursively (i.e, get all files from subdirectories, etc.)*senablesstep generation, Each timerefyre.step()is called, the next directory in the pattern is generated. (ex:, ifdir="test$"and*s,test1would be generated on first.step(),test2, ...)*cenables code analysis. If you seek to import a directory / repository you recently cloned, you can specify the*cflag and then import it in your codetype & linkare used for specific behaviours, most commonlygit cloning. Automatically clone in repos by specifyingtype="git"and the link to your git.mode: Can either be set tocutorcopy. During generation, the variable files will either be cut or copied to their respective place.limit: Limits the number of results targeted, or directories generatedserializespecify a generator expression to rename all the files into a consistent formatThese are all the basic quantifiers you can use, they cover ~80% of refyre's inner power. The other 20% are pretty obscure and aren't that useful normally.FileClusters (Variables)Variables are the backbone of refyre. The clusters provide anavenuefor the variables to easily target the data without worrying about writing any code. However, they aren't the only way to access variable's powers. The docs below, again, specify the most useful abilities for these variables.FileCluster(values = [], dirs = [], patterns = [], as_pathlib = False,)-values: string filepaths, orPathobjects depending on whetheras_pathlibis true or false.
-patterns: corresponds to the dirs, lists what patterns you want to targetFileClusters are strongly rooted inobject oriented operations, meaning each operation returns another FileCluster, so you can continue channeling FileCluster capabilities. To get out of FileClusters, you can use the following options:
-.vals(): Returns a list of Path objects
-.item(): Returns the first Path objectUsing this basic constructor, you can make some easy operations:.move(target_dir).copy(target_dir).filter(filter_func).map(map_func).zip().delete().post(url, additional_data, payload_name).filesize().clone()You can also do other operations between FileClusters+(Returns the sum of the contents of two FileClusters)-(Returns the contents in the current FileCluster while removing all other contents that are also in the other FileCluster)&(Intersection operator)|(Union operator)The Refyre ObjectThese docs are running too long already, I will try to keep this as short as possible.Refyre(input_specs = [], output_specs = [])Instantiates a refyre specadd_spec(spec_path, track = False)Adds a spec for refyrereading. If track is set to true, it can
later be reused for step generation.create_spec(spec_path, track = False)Creates a spec. If track is set to true, it can later be reused for step
generation.step()Any specs with a*sattribute have the next directory in the patterns they specify generatedAccessing variables can be done using the[]notation. Use it to get and attach variables to aRefyreobject.Congratulations, you know everything to be a refyre expert!Misc DocsDataStackLet's say you want to brew a dataset & structure data of you're own. refyre allows you to combine the power of variables with the DataStack, processing them to create constructs such as Pandas Dataframes.The process is twofold - (1)secure your variables, and then (2)run them through the DataStack. The DataStack itself is aprocessor, taking in a bunch of variables, and producing the variables.Your job with the DataStack will be to figure out how can you convert the variables to the dataset format you want.Consider the PandasStack (a DataStack). Here, your job is to figure out how you can convert each row of variables into aDataFrame columnfromrefyreimportRefyrefromrefyre.datastackimportPandasStackfromPILimportImageimportpandasaspdref=Refyre(input_specs=['specs/in.txt'])#We will do some pandas visualizations on the input datastack=PandasStack(AssociationCluster(input_vars=[ref["images"]]))defprocessor(fp):print('processing',fp)im=Image.open(fp).convert('RGB')width,height=im.sizear,ag,ab=0.0,0.0,0.0foriinrange(width):forjinrange(height):r,g,b=im.getpixel((i,j))ar,ag,ab=ar+r,ag+g,ab+bar,ag,ab=ar/(width*height),ag/(width*height),ab/(width*height)return(fp.name,width,height,ar,ag,ab)df=stack.create_dataframe(['image_name','image_width','image_height','average_red','average_green','average_blue'],processor)As you can see, the majority of the work here comes from building aprocessormethod to convert each row of variables into a DataFrame row.