Package: dynamic-loader
DataLoaderTable of ContentsRequirementsGetting StartedInstallation MethodsCloning the RepositoryInstall via pipOverviewFeaturesDataLoaderKey FeaturesParametersClass & Property MethodsDataMetricsKey FeaturesParametersProperty MethodsExtensionsKey FeaturesParametersClass MethodsGetLoggerOverviewParametersAttributesReturnsNoteUsageDataLoader UsageLoad Files from a Single Directory as a GeneratorLoad Files from a Single Directory as a Dictionary (Custom-Repr)Load Files from Multiple DirectoriesLoad Files with Default ExtensionsRetrieve Data for a Specific FileLoad Files with Custom Loader MethodsSpecify a Custom LoggerDataMetrics UsageExtensions UsageGetLogger UsageOutput ExampleFuture UpdatesFeedbackContact InformationRequirementsPython: ~=3.10pytest: ~=7.4.3setuptools: ~=68.2.2pandas: ~=2.2.0This project mandates the use ofPython 3.7or later versions. Compatibility issues have been identified with the use for dataclasses inPython 3.6and earlier versions.Getting StartedInstallation MethodsCloning the RepositoryClone the repository.Install the required dependenciespipinstall-rrequirements.txtInstall via pippipinstalldynamic-loader
pipinstall-rrequirements.txtOverview.TheDataLoaderproject is a comprehensive utility that facilitates the efficient loading and processing of data from specified directories. This project is designed to be user-friendly and easy to integrate into your projects.TheDataMetricsclass focuses on processing data paths and gathering statistics related to the file system and specified paths. Also allows the ability to export all statistics to a JSON file.TheExtensionsclass is a utility that provides a set of default file extensions for theDataLoaderclass. Its the back-bone for mapping all file extensions to its respective loading method.FeaturesDataLoaderTheDataLoaderclass is specifically designed for loading and processing data from directories. It provides the following key features:Key Features:Dynamic Loading: Load files from a single directory or merge files from multiple directories.Flexible Configuration: Set various parameters, such as default file extensions, full POSIX paths, method loader execution, and more.Parallel Execution: Leverage parallel execution with thetotal_workersparameter to enhance performance.Verbose Output: Display verbose output to track the loading process.If enabled, theverboseparameter will display the loading process for each file.If disabled, theverboseparameter will write the loading process for each file to a log file.Custom Loaders: Implement custom loaders for specific file extensions.Please note that at the moment, the loading methods kwargs will be uniformly applied to all files with the specified extension.Additionally, the first parameter of the loader method is automatically passed and should be skipped. If passed, the loader will fail and return the contents of the file asTextIOWrapper.Future updates will include the ability to specify what loader method to use for a specific files efficiently.Parameters:path(str or Path): The path of the directory from which to load files.directories(Iterable): An iterable of directories from which to all files.default_extensions(Iterable): Default file extensions to be processed.full_posix(bool): Indicates whether to display full POSIX paths.no_method(bool): Indicates whether to skip loading method matching execution.verbose(bool): Indicates whether to display verbose output.generator(bool): Indicates whether to return the loaded files as a generator; otherwise, returns as a dictionary.total_workers(int): Number of workers for parallel execution.log(Logger): A configured logger instance for logging messages. (Refer to theGetLoggerclass for more information on how to create a logger instance using theGetLoggerclass.)ext_loaders(dict[str, Any, dict[key-value]]): Dictionary containing extensions mapped to specified loaders. (Refer to theExtensionsclass for more information)Class & Property Methods:load_file(class_method): Load a specific file.get_files(class_method): Retrieve files from a directory based on default extensions and filters unwanted files.dir_files(property): Loaded files from specified directories.files(property): Loaded files from a single directoryall_exts(property): Retrieve all supported file extensions with their respective loader methods being used.EXTENSIONS(Extensionsclass instance): Retrieve all default supported file extensions with their respective loader methods.DataMetricsTheDataMetricsclass focuses on processing data paths and gathering statistics related to the file system. Key features include:Key Features:OS Statistics: Retrieve detailed statistics for each path, including symbolic link status, calculated size, and size in bytes.Export to JSON: Export all statistics to a JSON file for further analysis and visualization.Parameters:paths(Iterable): Paths for which to gather statistics.file_name(str): The file name to be used when exporting all files metadata stats.full_posix(bool): Indicates whether to display full POSIX paths.Property Methods:all_stats: Retrieve statistics for all paths.total_size: Calculate the total size of all paths.total_files: Calculate the total number of files in all paths.export_stats(): Export all statistics to a JSON file.OS Stats Results:os_stats_results: OS statistics results for each path.Custom Stats:st_fsize: Full file size statistics.st_vsize: Full volume size statistics.Extensions:TheExtensionsclass is a utility that provides a set of default file extensions for theDataLoaderclass. Its the back-bone for mapping all file extensions to its respective loading method. All extensions are stored in a dictionary (no period included), and theExtensionsclass provides the following key features:Key Features:File Extension Mapping: Retrieve all supported file extensions with their respective loader methods.Loader Method Retrieval: Retrieve the loader method for a specific file extension.Loader Method Check: Check if a specific file extension has a loader method implemented that's notopen.Supported Extension Check: Check if a specific file extension is supported.Customization: Customize theExtensionsclass with new files extensions and its respective loader methods.Parameters:No parameters are required for theExtensionsclass.Extensions(): Initializes theExtensionsclass with all implemented file extensions and their respective loader methods.Acts as a dictionary for accessing supported file extensions and their loader methods via Extensions().ALL_EXTS.Class Methods:ALL_EXTS: Retrieve all supported file extensions with their respective loader methods.get_loader: Retrieve the loader method for a specific file extension.has_loader: Checks if a specific file extension has a loader method implemented thats notopen.is_supported: Checks if a specific file extension is supported.customize: Customize theExtensionsclass with new files extensions and its respective loader methods.Specified loading method will be converted to a lambda function to support kwargs.The first parameter of the loader method is automatically passed and should be skipped. If passed, the loader will fail and return the contents of the file asTextIOWrapper.Future updates will include the ability to specify what loader method to use for a specific files efficiently.The loader method kwargs will be uniformly applied to all files with the specified extension.Example:# Structure: {extension: {loader_method: {kwargs}}}ext_loaders={"csv":{pd.read_csv:{"header":10}}}GetLoggerOverviewTheGetLoggerclass is a utility that provides a method to get a configured logger instance for logging messages. It is designed to be user-friendly and easy to integrate into your projects.Parametersname(str, optional): The name of the logger. Defaults to the name of the calling module.level(int, optional): The logging level. Defaults to logging.DEBUG.formatter_kwgs(dict, optional): Additional keyword arguments for the log formatter.handler_kwgs(dict, optional): Additional keyword arguments for the log handler.mode(str, optional): The file mode for opening the log file. Defaults to "a" (append).Attributesrefresher(callable): A method to refresh the log file.set_verbose(callable): A method to set the verbosity of the logger.ReturnsLogger: A configured logger instance.NotesThis function sets up a logger with a file handler and an optional stream (console) handler for verbose logging.Ifverboseis True, log messages will be printed to the console instead of being written to a file.Usage:DataLoaderUsage ExamplesLoad Files from a Single Directory as a Generatorfromdata_loaderimportDataLoader# Load all files with a specified path (directory) as a Generatordl_gen=DataLoader(path="path/to/directory")dl_files_gen=dl_gen.filesprint(dl_files_gen)# Output:# <generator object DataLoader.files.<key-value> at 0x1163f4ba0>Load Files from a Single Directory as a Dictionary (Custom-Repr)fromdata_loaderimportDataLoader# Load all files with a specified path (directory) as a Dictionary (Custom-Repr)# Disabling 'generator' and 'full_posix' for displaying purposes.dl_dict=DataLoader(path="path/to/directory",generator=False,full_posix=False)dl_files_dict=dl_dict.filesprint(dl_files_dict)# Output:# DataLoader((LICENSE.md, <TextIOWrapper>),#             (requirements.txt, <Str>),#             (Makefile, <Str>),#             ...#             (space_4.txt, <Str>))Load Files from Multiple Directoriesfromdata_loaderimportDataLoader# Load all files from multiple directories# Disabling 'generator' and 'full_posix' for displaying purposes.dl=DataLoader(directories=["path/to/dir1","path/to/dir2"],generator=False,full_posix=False)dl_dir_files=dl.dir_filesprint(dl_dir_files)# Output:# DataLoader((file1.txt, <Str>),#             (file2.txt, <Str>),#             (file3.txt, <Str>),#             ...#             (fileN.txt, <Str>))Load Files with Default Extensionsfromdata_loaderimportDataLoader# Load all files with default extensionsdl_default=DataLoader(path="path/to/directory",default_extensions=["csv"],generator=False,full_posix=False)dl_default_files=dl_default.filesprint(dl_default_files)# Output:# DataLoader((file1.csv, <DataFrame>),#             (file2.csv, <DataFrame>),#             ...#             (fileN.csv, <DataFrame>))Retrieve Data for a Specific Filefromdata_loaderimportDataLoader# Retrieve data for a specific filedl_files=DataLoader(path="path/to/directory",generator=False,full_posix=False).filesdl_specific_file_data=dl_files["file1.csv"]# Output:# <DataFrame>Load Files with Custom Loader Methodsfromdata_loaderimportDataLoaderimportpandasaspd# Specify your own custom loader methodsdl_custom=DataLoader(path="path/to/directory",ext_loaders={"csv":{pd.read_csv:{"nrows":10}}},generator=False,full_posix=False)dl_custom_files=dl_custom.filesprint(dl_custom_files)# Output:# DataLoader((file1.csv, <DataFrame>),#             (file2.csv, <DataFrame>),#             ...#             (fileN.csv, <DataFrame>))# Note: The 'nrows' will be dynamically passed to the 'pd.read_csv' method for each file.Specify a Custom Loggerfromdata_loaderimportDataLoaderimportlogging# Specify your own custom loggercustom_logger=logging.getLogger("DataLoader")dl_with_logger=DataLoader(path="path/to/directory",log=custom_logger)dl_logger_files=dl_with_logger.filesprint(dl_logger_files)# Output:# <generator object DataLoader.files.<key-value> at 0x1163f4ba0># Note: The logger will be used to log or stream messages.DataMetricsUsagefromdata_loaderimportDataMetrics# Retrieve statistics for all pathsdm=DataMetrics(files=["path/to/directory1","path/to/directory2"])print(dm.all_stats)# Retrieve statistics for all paths# Calculate the total size of all pathsprint(dm.total_size)# Calculate the total size of all paths# Calculate the total number of files in all pathsprint(dm.total_files)# Calculate the total number of files in all pathsdm.export_stats()# Export all statistics to a JSON fileExtensionsUsagefromdata_loaderimportExtensionsALL_EXTS=Extensions()# Initializes the Extensions class or use the default instance Extensions().ALL_EXTSprint("csv"inALL_EXTS)# Trueprint(ALL_EXTS.get_loader("csv"))# <function read_csv at 0x7f8e3e3e3d30># orprint(ALL_EXTS.get_loader(".pickle"))# <function read_csv at 0x7f8e3e3e3d30>print(ALL_EXTS.has_loader("docx"))# Falseprint(ALL_EXTS.is_supported("docx"))# TrueALL_EXTS.customize({"docx":{open:{mode="rb"}},"png":{PIL.Image.open:{}}})# Customize the Extensions class with a new file extension and loader methodprint(ALL_EXTS.get_loader("docx"))# <function <lambda> at 0x7f8e3e3e3d30>GetLoggerUsage:# Create a logger with default settingsfromdata_loaderimportGetLoggerlogger=GetLogger().loggerlogger.info("This is an info message")# Writes to the log file# Create a logger with custom settingslogger=GetLogger(name='custom_logger',level=logging.INFO,verbose=True).loggerlogger.info("This is an info message")# Prints to the console# Initiate verbositylogger=GetLogger().loggerlogger.set_verbose(True)CustomException("Error Message")# Prints to the console# Disable verbositylogger.set_verbose(False).loggerCustomException("Error Message")# Writes to the log fileDataMetricsUsage Examplesfromdata_metricsimportDataMetrics# Create a DataMetrics instance with paths and corresponding metadatadm=DataMetrics(("path/to/directory1",<Dict>),("path/to/directory2",<Dict>))# Access metadata for a specific pathmetadata_directory1=dm["path/to/directory1"]print(metadata_directory1)# Output:# {'os_stats_results': <os_stats_results>,#  'st_fsize': Stats(symbolic='6.20 KB', calculated_size=6.19921875, bytes_size=6348),#  'st_vsize': {'total': Stats(symbolic='465.63 GB (Gigabytes)', calculated_size=465.62699127197266, bytes_size=499963174912),#               'used': Stats(symbolic='131.60 GB (Gigabytes)', calculated_size=131.59552001953125, bytes_size=141299613696),#               'free': Stats(symbolic='334.03 GB (Gigabytes)', calculated_size=334.0314712524414, bytes_size=358663561216)}}# Export all statistics to a JSON filedm.export_stats(file_path="all_metadata_stats.json")# Calculate the total size of all pathstotal_size=dm.total_sizeprint(total_size)# Output:# Stats(symbolic='471.76 GB (Gigabytes)', calculated_size=471.75720977783203, bytes_size=507012679260)# Calculate the total number of files in all pathstotal_files=dm.total_filesprint(total_files)# Output:# 215Future UpdatesInclude the ability to specify loader methods for individual files, providing greater flexibility.Intend to add an option for special representation of loaded files, displaying all contents rather than just the data type.Add more comprehensive tests covering all implemented features.Include specific tests for theext_loadersparameter.Add loading method keyword argument support for theload_fileclass method.Implement a more efficient method for specifying loader methods kwargs for specific files rather than applying them uniformly.FeedbackFeedback is crucial for the improvement of theDataLoaderproject. If you encounter any issues, have suggestions, or want to share your experience, please consider the following channels:GitHub Issues: Open an issue on theGitHub repositoryto report bugs or suggest enhancements.Contact: Reach out to the project maintainer via the following:Contact InformationDiscordGmailYour feedback and contributions play a significant role in making theDataLoaderproject more robust and valuable for the community. Thank you for being part of this endeavor!