Package: scify-file-reader
scify-file-readerThe scify-file-reader package provides a convenient class for handling multiple files with the same structure in a directory. It offers functionality to read and process data from various file types, including CSV, XLSX, Parquet, and JSON.InstallationYou can install scify-file-reader using pip:pipinstallscify-file-readerUsageTo use scify-file-reader, follow these steps:Import theFileReaderclass:fromscify_file_readerimportFileReaderCreate an instance of the FileReader class, providing the content you want to read. The content can be a string representing afile path, aPathobject, or azipfile.ZipFileobject:content='path/to/directory'reader=FileReader(content)Read the files using the read_files method:data=reader.read_files()Theread_filesmethod returns a dictionary where the keys are the filenames (without the extension) and the values are pandas DataFrames containing the file data.For more details on the available methods and parameters, refer to the package documentation.Examples:Here's an example that demonstrates how to use scify-file-reader:Normal Outputfromscify_file_readerimportFileReaderPATH='/path/to/directory'"""# Supomos que temos estes arquivos dentro do nosso diretórioprint(os.listdir(PATH))# OUT: ['file_1.csv'', 'log_2.csv', 'test_3.csv','file_%Y%m%d%H%M%S.csv', 'log_%Y%m%d%H%M%S.csv', 'test_%Y%m%d%H%M%S.csv','file_%Y%m%d_%H%M%S.csv', 'log_%Y%m%d_%H%M%S.csv', 'test_%Y%m%d_%H%M%S.csv',"""# Example: Reading files from a directoryreader=FileReader('/path/to/directory')data=reader.read_files()# read_files accept kwargs from pandas read_ methods"""OUTPUT: print(data){'file_1.csv': <pd.DataFrame>,'log_2.csv': <pd.DataFrame>,'test_3.csv': <pd.DataFrame>,'file_%Y%m%d%H%M%S.csv': <pd.DataFrame>,'log_%Y%m%d%H%M%S.csv': <pd.DataFrame>,'test_%Y%m%d%H%M%S.csv': <pd.DataFrame>,'file_%Y%m%d_%H%M%S.csv': <pd.DataFrame>,'log_%Y%m%d_%H%M%S.csv': <pd.DataFrame>,'test_%Y%m%d_%H%M%S.csv': <pd.DataFrame>}"""Concatenating patterns:Use this method when you need to concatenate multiple files with similar patterns into a single consolidated file.E.g.In the last example, we demonstrate the use of scify-file-reader with a directory containing 9 files that follow common naming patterns, such as 'file', 'log', and 'test'. By joining these files, we can consolidate and analyze their data more effectively. Let's take a look at the example to understand how they are joined.fromscify_file_readerimportFileReaderPATH='/path/to/directory'"""# Let's suppose we have these files inside our directory.print(os.listdir(PATH))# OUT: ['file_1.csv'', 'log_2.csv', 'test_3.csv','file_%Y%m%d%H%M%S.csv', 'log_%Y%m%d%H%M%S.csv', 'test_%Y%m%d%H%M%S.csv','file_%Y%m%d_%H%M%S.csv', 'log_%Y%m%d_%H%M%S.csv', 'test_%Y%m%d_%H%M%S.csv',"""# Example: Reading files from a directoryreader=FileReader('/path/to/directory')data=reader.read_files(join_prefixes=True)#"""OUTPUT: print(data){'file': <pd.DataFrame>,'log': <pd.DataFrame>,'test': <pd.DataFrame>,}"""Using a specific regular expressionIn the example above, all files with common prefixes, such asfile_1.csv,file_%Y%m%d%H%M%S.csv, andfile_%Y%m%d_%H%M%S.csv, were joined together under the file key in the output.If you want to use a specific regular expression for filtering your files, you can follow these steps:fromscify_file_readerimportFileReaderPATH='/path/to/directory'# Example: Reading files from a directoryreader=FileReader('/path/to/directory')regex='<some_regex>'reader.set_prefix_file_pattern_regex(regex)data=reader.read_files(join_prefixes=True)By default the regular expression is^([A-Z]+)_\d+.Speficic prefixes instead of regular expressionsIf you prefer to use specific prefixes instead of regular expressions, you can utilize thejoin_custom_prefixesargument. This argument accepts a tuple of prefixes that you want to join together.fromscify_file_readerimportFileReaderPATH='/path/to/directory'"""# Supomos que temos estes arquivos dentro do nosso diretórioprint(os.listdir(PATH))# OUT: ['file_1.csv'', 'log_2.csv', 'test_3.csv','file_%Y%m%d%H%M%S.csv', 'log_%Y%m%d%H%M%S.csv', 'test_%Y%m%d%H%M%S.csv','file_%Y%m%d_%H%M%S.csv', 'log_%Y%m%d_%H%M%S.csv', 'test_%Y%m%d_%H%M%S.csv',"""# Example: Reading files from a directoryreader=FileReader('/path/to/directory')specific_prefixes=('file','log','test')data=reader.read_files(join_prefixes=True)"""OUTPUT: print(data){'file': <pd.DataFrame>,'log': <pd.DataFrame>,'test': <pd.DataFrame>,}"""ContributingContributions are welcome! If you have any suggestions, bug reports, or feature requests, please open an issue or submit a pull request on thescify-file-readerrepository.