Package: package-auto-assembler
Package Auto AssemblerThis tool is meant to streamline creation of single module packages.
Its purpose is to automate as many aspects of python package creation as possible,
to shorten a development cycle of reusable components, maintain certain standard of quality
for reusable code. It provides tool to simplify the process of package creatrion
to a point that it can be triggered automatically within ci/cd pipelines,
with minimal preparations and requirements for new modules.importsyssys.path.append('../')frompython_modules.package_auto_assemblerimport(VersionHandler,\ImportMappingHandler,RequirementsHandler,MetadataHandler,\LocalDependaciesHandler,LongDocHandler,SetupDirHandler,\PackageAutoAssembler)Usage examplesThe examples contain:package versioningimport mappingextracting and merging requirementspreparing metadatamerging local dependacies into single moduleprepare READMEassembling setup directorymaking a package1. Package versioningInitialize VersionHandlerpv=VersionHandler(# requiredversions_filepath='../tests/package_auto_assembler/lsts_package_versions.yml',log_filepath='../tests/package_auto_assembler/version_logs.csv',# optionaldefault_version="0.0.1")Add new packagepv.add_package(package_name="new_package",# optionalversion="0.0.1")Update package versionpv.increment_patch(package_name="new_package")## for not tracked packagepv.increment_patch(package_name="another_new_package",# optionaldefault_version="0.0.1")There are no known versions of 'another_new_package', 0.0.1 will be used!Display current versions and logspv.get_versions(# optionalversions_filepath='../tests/package_auto_assembler/lsts_package_versions.yml'){'another_new_package': '0.0.1',
 'example_module': '0.0.1',
 'new_package': '0.0.2'}pv.get_version(package_name='new_package')'0.0.2'pv.get_logs(# optionallog_filepath='../tests/package_auto_assembler/version_logs.csv')<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}</style>TimestampPackageVersion02024-01-06 00:54:04example_module0.0.112024-02-17 18:55:39new_package0.0.122024-02-17 18:55:39new_package0.0.232024-02-17 18:55:39another_new_package0.0.1Flush versions and logspv.flush_versions()pv.flush_logs()2. Import mappingInitialize ImportMappingHandlerim=ImportMappingHandler(# requiredmapping_filepath="../env_spec/package_mapping.json")Load package mappingsim.load_package_mappings(# optionalmapping_filepath="../env_spec/package_mapping.json"){'PIL': 'Pillow',
 'bs4': 'beautifulsoup4',
 'fitz': 'PyMuPDF',
 'attr': 'attrs',
 'dotenv': 'python-dotenv',
 'googleapiclient': 'google-api-python-client',
 'google_auth_oauthlib': 'google-auth-oauthlib',
 'sentence_transformers': 'sentence-transformers',
 'flask': 'Flask',
 'stdlib_list': 'stdlib-list',
 'sklearn': 'scikit-learn',
 'yaml': 'pyyaml',
 'package_auto_assembler': 'package-auto-assembler'}3. Extracting and merging requirementsInitialize RequirementsHandlerrh=RequirementsHandler(# optional/required latermodule_filepath="../tests/package_auto_assembler/example_module.py",package_mappings={'PIL':'Pillow','bs4':'beautifulsoup4','fitz':'PyMuPDF','attr':'attrs','dotenv':'python-dotenv','googleapiclient':'google-api-python-client','sentence_transformers':'sentence-transformers','flask':'Flask','stdlib_list':'stdlib-list','sklearn':'scikit-learn','yaml':'pyyaml'},requirements_output_path="../tests/package_auto_assembler/",output_requirements_prefix="requirements_",custom_modules_filepath="../tests/package_auto_assembler/dependancies",python_version='3.8')List custom modules for a given directoryrh.list_custom_modules(# optionalcustom_modules_filepath="../tests/package_auto_assembler/dependancies")['example_local_dependacy_2', 'example_local_dependacy_1']Check if module is a standard python libraryrh.is_standard_library(# requiredmodule_name='example_local_dependacy_1',# optionalpython_version='3.8')Falserh.is_standard_library(# requiredmodule_name='logging',# optionalpython_version='3.8')TrueExtract requirements from the module filerh.extract_requirements(# optionalmodule_filepath="../tests/package_auto_assembler/example_module.py",custom_modules=['example_local_dependacy_2','example_local_dependacy_1'],package_mappings={'PIL':'Pillow','bs4':'beautifulsoup4','fitz':'PyMuPDF','attr':'attrs','dotenv':'python-dotenv','googleapiclient':'google-api-python-client','sentence_transformers':'sentence-transformers','flask':'Flask','stdlib_list':'stdlib-list','sklearn':'scikit-learn','yaml':'pyyaml'},python_version='3.8')['### example_module.py', 'attrs>=22.2.0']Save requirements to a filerh.write_requirements_file(# optional/required latermodule_name='example_module',requirements=['### example_module.py','attrs>=22.2.0'],output_path="../tests/package_auto_assembler/",prefix="requirements_")Read requirementsrh.read_requirements_file(# requiredrequirements_filepath="../tests/package_auto_assembler/requirements_example_module.txt")['attrs>=22.2.0']4. Preparing metadataInitializing MetadataHandlermh=MetadataHandler(# optional/required latermodule_filepath="../tests/package_auto_assembler/example_module.py")Check if metadata is availablemh.is_metadata_available(# optionalmodule_filepath="../tests/package_auto_assembler/example_module.py")TrueExtract metadata from modulemh.get_package_metadata(# optionalmodule_filepath="../tests/package_auto_assembler/example_module.py"){'author': 'Kyrylo Mordan',
 'author_email': 'parachute.repo@gmail.com',
 'version': '0.0.1',
 'description': 'A mock handler for simulating a vector database.',
 'keywords': ['python', 'vector database', 'similarity search']}5. Merging local dependacies into single moduleInitializing LocalDependaciesHandlerldh=LocalDependaciesHandler(# requiredmain_module_filepath="../tests/package_auto_assembler/example_module.py",dependencies_dir="../tests/package_auto_assembler/dependancies/",# optionalsave_filepath="./combined_example_module.py")Combine main module with dependaciesprint(ldh.combine_modules(# optionalmain_module_filepath="../tests/package_auto_assembler/example_module.py",dependencies_dir="../tests/package_auto_assembler/dependancies/",add_empty_design_choices=False)[0:1000])"""
Mock Vector Db Handler

This class is a mock handler for simulating a vector database, designed primarily for testing and development scenarios.
It offers functionalities such as text embedding, hierarchical navigable small world (HNSW) search,
and basic data management within a simulated environment resembling a vector database.
"""

import logging
import json
import time
import attr #>=22.2.0
import string
import os
import csv

__design_choices__ = {}

@attr.s
class Shouter:

    """
    A class for managing and displaying formatted log messages.

    This class uses the logging module to create and manage a logger
    for displaying formatted messages. It provides a method to output
    various types of lines and headers, with customizable message and
    line lengths.
    """

    # Formatting settings
    dotline_length = attr.ib(default=50)

    # Logger settings
    logger = attr.ib(default=None)
    logger_name = attr.ib(default='Shouter')
    loggerLvl = attr.ib(default=loSave combined moduleldh.save_combined_modules(# optionalcombined_module=ldh.combine_modules(),save_filepath="./combined_example_module.py")6. Prepare READMEimportloggingldh=LongDocHandler(# optional/required laternotebook_path="../tests/package_auto_assembler/example_module.ipynb",markdown_filepath="../example_module.md",timeout=600,kernel_name='python3',# loggerloggerLvl=logging.DEBUG)Convert notebook to md without executingldh.convert_notebook_to_md(# optionalnotebook_path="../tests/package_auto_assembler/example_module.ipynb",output_path="../example_module.md")Converted ../tests/package_auto_assembler/example_module.ipynb to ../example_module.mdConvert notebook to md with executingldh.convert_and_execute_notebook_to_md(# optionalnotebook_path="../tests/package_auto_assembler/example_module.ipynb",output_path="../example_module.md",timeout=600,kernel_name='python3')Converted and executed ../tests/package_auto_assembler/example_module.ipynb to ../example_module.mdReturn long descriptionlong_description=ldh.return_long_description(# optionalmarkdown_filepath="../example_module.md")7. Assembling setup directoryInitializing SetupDirHandlersdh=SetupDirHandler(# requiredmodule_filepath="../tests/package_auto_assembler/example_module.py",# optional/ requiredmodule_name="example_module",metadata={'author':'Kyrylo Mordan','version':'0.0.1','description':'Example module.','long_description':long_description,'keywords':['python']},requirements=['attrs>=22.2.0'],classifiers=['Development Status :: 3 - Alpha','Intended Audience :: Developers','Intended Audience :: Science/Research','Programming Language :: Python :: 3','Programming Language :: Python :: 3.9','Programming Language :: Python :: 3.10','Programming Language :: Python :: 3.11','License :: OSI Approved :: MIT License','Topic :: Scientific/Engineering'],setup_directory="./example_setup_dir")Create empty setup dirsdh.flush_n_make_setup_dir(# optionalsetup_directory="./example_setup_dir")Copy module to setup dirsdh.copy_module_to_setup_dir(# optionalmodule_filepath="./combined_example_module.py",setup_directory="./example_setup_dir")Create init filesdh.create_init_file(# optionalmodule_name="example_module",setup_directory="./example_setup_dir")Create setup filesdh.write_setup_file(# optionalmodule_name="example_module",metadata={'author':'Kyrylo Mordan','version':'0.0.1','description':'Example Module','keywords':['python']},requirements=['attrs>=22.2.0'],classifiers=['Development Status :: 3 - Alpha','Intended Audience :: Developers','Intended Audience :: Science/Research','Programming Language :: Python :: 3','Programming Language :: Python :: 3.9','Programming Language :: Python :: 3.10','Programming Language :: Python :: 3.11','License :: OSI Approved :: MIT License','Topic :: Scientific/Engineering'],setup_directory="./example_setup_dir")8. Making a packageInitializing PackageAutoAssemblerpaa=PackageAutoAssembler(# requiredmodule_name="example_module",module_filepath="../tests/package_auto_assembler/example_module.py",# optionalmapping_filepath="../env_spec/package_mapping.json",dependencies_dir="../tests/package_auto_assembler/dependancies/",example_notebook_path="./mock_vector_database.ipynb",versions_filepath='../tests/package_auto_assembler/lsts_package_versions.yml',log_filepath='../tests/package_auto_assembler/version_logs.csv',setup_directory="./example_module",classifiers=['Development Status :: 3 - Alpha','Intended Audience :: Developers','Intended Audience :: Science/Research','Programming Language :: Python :: 3','Programming Language :: Python :: 3.9','Programming Language :: Python :: 3.10','Programming Language :: Python :: 3.11','License :: OSI Approved :: MIT License','Topic :: Scientific/Engineering'],requirements_list=[],execute_readme_notebook=True,python_version="3.8",version_increment_type="patch",default_version="0.0.1")Add metadata from modulepaa.add_metadata_from_module(# optionalmodule_filepath="../tests/package_auto_assembler/example_module.py")Add or update versionpaa.add_or_update_version(# optionalmodule_name="example_module",version_increment_type="patch",version="0.0.1",versions_filepath='../tests/package_auto_assembler/lsts_package_versions.yml',log_filepath='../tests/package_auto_assembler/version_logs.csv')There are no known versions of 'example_module', 0.0.1 will be used!Prepare setup directorypaa.prep_setup_dir()Merge local dependaciespaa.merge_local_dependacies(# optionalmain_module_filepath="../tests/package_auto_assembler/example_module.py",dependencies_dir="../tests/package_auto_assembler/dependancies/",save_filepath="./example_module/example_module.py")Add requirements from modulepaa.add_requirements_from_module(# optionalmodule_filepath="../tests/package_auto_assembler/example_module.py",import_mappings={'PIL':'Pillow','bs4':'beautifulsoup4','fitz':'PyMuPDF','attr':'attrs','dotenv':'python-dotenv','googleapiclient':'google-api-python-client','sentence_transformers':'sentence-transformers','flask':'Flask','stdlib_list':'stdlib-list','sklearn':'scikit-learn','yaml':'pyyaml'})Make README out of example notebookpaa.add_readme(# optionalexample_notebook_path="../tests/package_auto_assembler/example_module.ipynb",output_path="./example_module/README.md",execute_notebook=True,)paa.requirements_list['### example_module.py', 'attrs>=22.2.0']Prepare setup filepaa.prep_setup_file(# optionalmetadata={'author':'Kyrylo Mordan','version':'0.0.1','description':'Example module','keywords':['python']},requirements=['### example_module.py','attr>=22.2.0'],classifiers=['Development Status :: 3 - Alpha','Intended Audience :: Developers','Intended Audience :: Science/Research','Programming Language :: Python :: 3','Programming Language :: Python :: 3.9','Programming Language :: Python :: 3.10','Programming Language :: Python :: 3.11','License :: OSI Approved :: MIT License','Topic :: Scientific/Engineering'])Make packagepaa.make_package(# optionalsetup_directory="./example_module")CompletedProcess(args=['python', './example_module/setup.py', 'sdist', 'bdist_wheel'], returncode=0, stdout="running sdist\nrunning egg_info\ncreating example_module.egg-info\nwriting example_module.egg-info/PKG-INFO\nwriting dependency_links to example_module.egg-info/dependency_links.txt\nwriting requirements to example_module.egg-info/requires.txt\nwriting top-level names to example_module.egg-info/top_level.txt\nwriting manifest file 'example_module.egg-info/SOURCES.txt'\nreading manifest file 'example_module.egg-info/SOURCES.txt'\nwriting manifest file 'example_module.egg-info/SOURCES.txt'\nrunning check\ncreating example_module-0.0.1\ncreating example_module-0.0.1/example_module\ncreating example_module-0.0.1/example_module.egg-info\ncopying files to example_module-0.0.1...\ncopying example_module/__init__.py -> example_module-0.0.1/example_module\ncopying example_module/example_module.py -> example_module-0.0.1/example_module\ncopying example_module/setup.py -> example_module-0.0.1/example_module\ncopying example_module.egg-info/PKG-INFO -> example_module-0.0.1/example_module.egg-info\ncopying example_module.egg-info/SOURCES.txt -> example_module-0.0.1/example_module.egg-info\ncopying example_module.egg-info/dependency_links.txt -> example_module-0.0.1/example_module.egg-info\ncopying example_module.egg-info/requires.txt -> example_module-0.0.1/example_module.egg-info\ncopying example_module.egg-info/top_level.txt -> example_module-0.0.1/example_module.egg-info\nWriting example_module-0.0.1/setup.cfg\ncreating dist\nCreating tar archive\nremoving 'example_module-0.0.1' (and everything under it)\nrunning bdist_wheel\nrunning build\nrunning build_py\ncreating build\ncreating build/lib\ncreating build/lib/example_module\ncopying example_module/example_module.py -> build/lib/example_module\ncopying example_module/__init__.py -> build/lib/example_module\ncopying example_module/setup.py -> build/lib/example_module\ninstalling to build/bdist.linux-x86_64/wheel\nrunning install\nrunning install_lib\ncreating build/bdist.linux-x86_64\ncreating build/bdist.linux-x86_64/wheel\ncreating build/bdist.linux-x86_64/wheel/example_module\ncopying build/lib/example_module/example_module.py -> build/bdist.linux-x86_64/wheel/example_module\ncopying build/lib/example_module/__init__.py -> build/bdist.linux-x86_64/wheel/example_module\ncopying build/lib/example_module/setup.py -> build/bdist.linux-x86_64/wheel/example_module\nrunning install_egg_info\nCopying example_module.egg-info to build/bdist.linux-x86_64/wheel/example_module-0.0.1-py3.10.egg-info\nrunning install_scripts\ncreating build/bdist.linux-x86_64/wheel/example_module-0.0.1.dist-info/WHEEL\ncreating 'dist/example_module-0.0.1-py3-none-any.whl' and adding 'build/bdist.linux-x86_64/wheel' to it\nadding 'example_module/__init__.py'\nadding 'example_module/example_module.py'\nadding 'example_module/setup.py'\nadding 'example_module-0.0.1.dist-info/METADATA'\nadding 'example_module-0.0.1.dist-info/WHEEL'\nadding 'example_module-0.0.1.dist-info/top_level.txt'\nadding 'example_module-0.0.1.dist-info/RECORD'\nremoving build/bdist.linux-x86_64/wheel\n", stderr='warning: sdist: standard file not found: should have one of README, README.rst, README.txt, README.md\n\n/opt/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.\n  warnings.warn(\n')