Package: ml-indie-tools
A collection of machine learning tools for low-resource research and experimentsDescriptionpipinstallml-indie-toolsThis module contains of a collection of tools useable for researchers with limited access to compute-resources and
who change between laptop, Colab-instances and local workstations with a graphics card.env_toolschecks the current environment, and populates a number of flags that allow identification of run-time
environment and available accelerator hardware. For Colab instances, it provides tools to mount Google Drive for
persistent data- and model-storage.The usage scenarios are:EnvTensorflow TPUTensorflow GPUPytorch TPUPytorch GPUJax TPUJax GPUColab++/+++Workstation with Nvidia/+/+/+Apple Silicon/+/+//(+: supported,/: not supported)Gutenberg_DatasetandText_Datasetare NLP libraries that provide text data and can be used in conjuction
with HuggingfaceDatasetsor directly with ML libraries.ALU_Datasetis a toy-dataset that allows training of integer arithmetic and logical (ALU) operations.env_toolsA collection of tools that allow moving machine learning projects between local hardware and colab instances.ExamplesLocal laptop:fromml_indie_tools.env_toolsimportMLEnvml_env=MLEnv(platform='tf',accelator='fastest')ml_env.describe()# -> 'OS: Darwin, Python: 3.9.9 (Conda) Tensorflow: 2.7.0, GPU: METAL'ml_env.is_gpu# -> Trueml_env.is_tensorflow# -> Trueml_env.gpu_type# -> 'METAL'Colab instance:# !pip install -U ml_indie_toolsfromml_indie_tools.env_toolsimportMLEnvml_env=MLEnv(platform='tf',accelerator='fastest')print(ml_env.describe())print(ml_env.gpu_type)Output:DEBUG:MLEnv:Tensorflow version: 2.7.0
DEBUG:MLEnv:GPU available
DEBUG:MLEnv:You are on a Jupyter instance.
DEBUG:MLEnv:You are on a Colab instance.
INFO:MLEnv:OS: Linux, Python: 3.7.12, Colab Jupyter Notebook Tensorflow: 2.7.0, GPU: Tesla K80
The tensorboard extension is already loaded. To reload it, use:
  %reload_ext tensorboard
OS: Linux, Python: 3.7.12, Colab Jupyter Notebook Tensorflow: 2.7.0, GPU: Tesla K80
Tesla K80Project pathsml_env.init_paths('my_project', 'my_model')will give a list of paths that are adapted for local and colab usageLocal project:ml_env.init_paths("my_project","my_model")# -> ('.', '.', './model/my_model', './data', './logs')The list contains<root-path>,<project-path>(both are., the current directory for local projects),<model-path>to save model and weights,<data-path>for
training data and<log-path>for logs.Those paths (with exception of./logs) are moved to Google Drive for Colab instances:On Google Colab:# INFO:MLEnv:You will now be asked to authenticate Google Drive access in order to store training data (cache) and model state.
# INFO:MLEnv:Changes will only happen within Google Drive directory `My Drive/Colab Notebooks/<project-name>`.
# DEBUG:MLEnv:Root path: /content/drive/My Drive
# Mounted at /content/drive
('/content/drive/My Drive',
 '/content/drive/My Drive/Colab Notebooks/my_project',
 '/content/drive/My Drive/Colab Notebooks/my_project/model/my_model',
 '/content/drive/My Drive/Colab Notebooks/my_project/data',
 './logs')See theenv_tools API documentationfor details.Gutenberg_DatasetGutenberg_Dataset makes books fromProject Gutenbergavailable as dataset.This module can either work with a local mirror of Project Gutenberg, or download files on demand.
Files that are downloaded are cached to prevent unnecessary load on Gutenberg's servers.Working with a local mirror of Project GutenbergIf you plan to use a lot of files (hundreds or more) from Gutenberg, a local mirror might be the best
solution. Have a look atProject Gutenberg's notes on mirrors.A mirror image suitable for this project can be made with:rsync-zarv--dry-run--prune-empty-dirs--del--include="*/"--include='*.'{txt,pdf,ALL}--exclude="*"aleph.gutenberg.org::gutenberg./gutenberg_mirrorIt's not mandatory to includepdf-files, since they are currently not used. Please review the--dry-runflag.Once a mirror of at least all of Gutenberg's*.txtfiles and of index-fileGUTINDEX.ALLhas been generated, it can be used via:fromml_indie_tools.Gutenberg_DatasetimportGutenberg_Datasetgd=Gutenberg_Dataset(root_url='./gutenberg_mirror')# Assuming this is the file-path to the mirror imageWorking without a remote mirrorfromml_indie_tools.Gutenberg_DatasetimportGutenberg_Datasetgd=Gutenberg_Dataset()# the default Gutenberg site is used. Alternative specify a specific mirror with `root_url=http://...`.Getting Gutenberg booksAfter using one of the two methods to instantiate thegdobject:gd.load_index()# load the index of booksThen get a list of books (array). Each entry is a dict with meta-data:search_resultis a list of dictionaries containing meta-data without the actual book-text.search_result=gd.search({'author':['kant','goethe'],'language':['german','english']})Insert the actual book text into the dictionaries. Note that download count islimitedif using a remote server.search_result=gd.insert_book_texts(search_result)# search_result entries now contain an additional field `text` with the filtered text of the book.importpandasaspddf=pd.DataFrame(search_result)# Display results as Pandas DataFramedfSee theGutenberg_Dataset API documentationfor details.Text_DatasetA library for character, word, or dynamical ngram tokenization.importlogginglogging.basicConfig(encoding='utf-8',level=logging.INFO)fromml_indie_tools.Gutenberg_DatasetimportGutenberg_Datasetfromml_indie_tools.Text_DatasetimportText_Datasetgd=Gutenberg_Dataset()gd.load_index()bl=gd.search({'title':['proleg','hermen'],'language':['english']})bl=gd.insert_book_texts(bl)foriinrange(len(bl)):print(bl[i]['title'])Prolegomena to the Study of Hegel's PhilosophyKant's ProlegomenaThe Cornish Fishermen's Watch Night and Other StoriesProlegomena to the History of IsraelLegge Prolegomenatl=Text_Dataset(bl)# bl contains a list of texts (books from Gutenberg)tl.source_highlight("If we write anything that contains parts of the sources, like: that is their motto, then a highlight will be applied.")INFO:Datasets:Loaded 5 textsIf we write anything t[4]hat contains[1]parts of the s[4]ources, like:that is t[1]heir motto[4], then a highlight will be a[1]pplied.Sources: Julius Wellhausen: Prolegomena to the History of Israel[4], William Wallace and G. W. F. Hegel: Prolegomena to the Study of Hegel's Philosophy[1]test_text="That would be a valid argument if we hadn't defeated it's assumptions way before."print(f"Text length{len(test_text)},{test_text}")tokenizer='ngram'tl.init_tokenizer(tokenizer=tokenizer)st=tl.tokenize(test_text)print(f"Token-count:{len(st)},{st}")Text length 81, That would be a valid argument if we hadn't defeated it's assumptions way before. Token-count: 27, [1447, 3688, 1722, 4711, 4880, 1210, 1393, 4393, 2382, 1352, 3655, 1972, 1939, 44, 23, 3333, 1871, 4975, 2967, 2884, 2216, 2382, 3048, 1546, 4589, 2272, 30]test2="ðƒ "+test_textprint(f"Text length{len(test2)},{test2}")el=tl.encode(test2)print(f"Token-count:{len(el)},{el}")Text length 84, ðƒ That would be a valid argument if we hadn't defeated it's assumptions way before. Token-count: 29, ['<unk>', '<unk>', 1397, 3688, 1722, 4711, 4880, 1210, 1393, 4393, 2382, 1352, 3655, 1972, 1939, 44, 23, 3333, 1871, 4975, 2967, 2884, 2216, 2382, 3048, 1546, 4589, 2272, 30]See theText_Dataset API documentationfor details.ALU_DatasetSee theALU_Dataset API documentationfor details.
A sample project is atALU_Netkeras_custom_layersA collection of Keras residual- and self-attention layersSee thekeras_custom_layers API documentationfor details.External projectsCheckout the following jupyter notebook based projects for example-usage:Text generationtensor-poettorch-poettransformer-poettorch-transformer-poet, using pytorch transformers from Andrej Karpathy's nanoGPT as implemented inng-video-lectureArithmetic and logic operationsALU_NetHistory(2024-02-21, 0.10.4) More tests with bytegrams.(2023-11-14, 0.9.3) Fix/hack for reloading of checkpoints for compiled models (torch puts weights in some sub-object:_orig_mod)(2023-11-13, 0.9.0) Breaking API change toFolder_Dataset,load_index()can be called multiple times additively.(2023-04-2, 0.8.168) Two transformer variants: a transformer with 'yoke',MultiHeadSelfAttentionWithCompression(a layer that compresses information, forcing abstraction), and version with state:MultiHeadSelfAttentionWithCompressionState, a state is combined with the yoke-layer, allowing the most 'abstract' information of the transformer to be maintained in a recurrent manner. Seetorch-transformer-poetfor examples.(2023-04-01, 0.8.90) API changes: WIP!(2023-03-31, 0.8.0) Put compression/state experiments in separate model.(2023-03-30, 0.7.0) Cleanup of bottleneck mechanism to force abstraction. Dropout behave again normal
(hacks removed).(2023-03-28, 0.6.0) Adddropout>1.0paramater to MultiHeadSelfAttention (torch): replaces 'normal' dropout with a linear compression by 4.0/dropout. The linear layers no longer
map n -> 4n -> n, but n -> 4n/dropout -> n. This reduces the amount of information, the net can propagate, forcing compression. Sigma_compression uses different compressions rates:
max in the middle layers, and non at start end end layers, linearly interpolating between them.(2023-02-01, 0.5.6)load_checkpoint(), optionally only loadparams. Incompatible API-change forload-andsave_checkpoint()methods!(2023-01-31, 0.5.4) Addtop_kparameter to generator. Apple MPS users beware, MPScurrently limits top_k to max 16.(2023-01-30, 0.5.3) Adduse_aliasesparameter to Folder- and Calibre datasets.(2023-01-27, 0.5.2) Addaliasfield to local datasets to protect
privacy of local document names.(2023-01-27, 0.5.0) Acquire training data from Calibre library (Calibre_Dataset), the documents must be in text format in Calibre, or get training data from a folder containing text files (Folder_Dataset). Text_Dataset can now
contain texts from Gutenberg, Calibre or a folder of text files.(2023-01-26, 0.4.4) Add save/load tokenizer to Text_Dataset to enable reusing tokenizer data.(2023-01-22, 0.4.3) Add temperature parameter to generator.(2023-01-21, 0.4.2) Start of port of pytorch transformers from Andrej Karpathy's nanoGPT as implemented inng-video-lecture. Additional tests with Apple Silicon MPS and pytorch 2.0 nightly.(2022-12-13, 0.4.0) The great cleanup: neither recurrence nor gated memory improved the transformer architecture, so they are removed again.(2022-12-11, 0.3.17) Testversion for slightly handwavy recurrent attention(2022-06-19, 0.3.1) get_random_item(index) that works with all tokenization strategies, get_unique_token_count() added.(2022-06-19, 0.3.0) Breaking change in Text_Datasetget_item() behavior, old API didn't fit with tokenization.(2022-06-19, 0.2.0) Language agnostic dynamic ngram tokenizer.(2022-06-07, 0.1.5) Support for pytorch nightly 1.13dev MPS, Apple Metal acceleration on Apple Silicon.(2022-03-27, 0.1.4) Bugfixes to Gutenbergsearchandload_bookandget_book.(2022-03-15, 0.1.2)env_tools.init()no longer usestf.compat.v1.disable_eager_executition()since there are rumors about old code-paths being used. Usetf.function()instead, or call withenv_tools.init(..., old_disable_eager=True)which continues to use the old v1 API.(2022-03-12, 0.1.0) First version for external use.(2021-12-26, 0.0.x) First pre-alpha versions published for testing purposes, not ready for use.