Repository: h1alexbel/samples-filter (language: python)

Issue:
Update models README



Discussion / hints:

Fix patch:
diff --git a/.github/workflows/docker-models.yml b/.github/workflows/docker-models.yml
new file mode 100644
index 0000000..aab6070
--- /dev/null
+++ b/.github/workflows/docker-models.yml
@@ -0,0 +1,47 @@
+# The MIT License (MIT)
+#
+# Copyright (c) 2024 Aliaksei Bialiauski
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included
+# in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+---
+name: docker-models
+'on':
+  push:
+    tags:
+      - '*'
+permissions:
+  contents: read
+jobs:
+  build:
+    runs-on: ubuntu-22.04
+    steps:
+      - uses: actions/checkout@v4
+      - name: Build
+        run: |
+          cd models
+          make install test
+      - name: Push to docker.io
+        uses: mr-smithers-excellent/docker-build-push@v6
+        with:
+          image: abialiauski/samples-filter-models
+          tags: ${{ github.ref_name }}, latest
+          dockerfile: Dockerfile
+          registry: docker.io
+          username: abialiauski
+          password: ${{ secrets.DOCKER_PASSWORD }}
diff --git a/models/.dockerignore b/models/.dockerignore
new file mode 100644
index 0000000..e15a381
--- /dev/null
+++ b/models/.dockerignore
@@ -0,0 +1,4 @@
+venv/
+*.egg-info/
+build/
+model_tests/
diff --git a/models/Dockerfile b/models/Dockerfile
new file mode 100644
index 0000000..f1dfa4f
--- /dev/null
+++ b/models/Dockerfile
@@ -0,0 +1,30 @@
+# The MIT License (MIT)
+#
+# Copyright (c) 2024 Aliaksei Bialiauski
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included
+# in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+FROM python:3.11-slim AS stage
+WORKDIR /app
+COPY . .
+RUN pip install --no-cache-dir -r requirements.txt
+RUN python3 tools.py
+FROM python:3.11-slim as distilled
+WORKDIR /app
+COPY --from=stage /app/model ./model
+ENTRYPOINT ["python3"]
diff --git a/models/Makefile b/models/Makefile
index 2e62d41..d114703 100644
--- a/models/Makefile
+++ b/models/Makefile
@@ -22,7 +22,7 @@
 
 .SHELLFLAGS: -e -o pipefail -c
 .ONESHELL:
-.PHONY: install test if transformer
+.PHONY: install test
 .SILENT:
 
 ## The shell to use.
@@ -38,15 +38,3 @@ install:
 test:
 	export PYTHONPATH=.
 	python3 -m pytest model_tests
-
-# Train model with isolation forest algorithm.
-# @todo #129:90min Train isolation forest model.
-#  We need to find out how to properly train isolation forest model with
-#  gathered data in order to detect anomalies (SRs). Don't forget to remove
-#  this puzzle.
-if:
-	echo "training model based on isolation forest algorithm..."
-
-# Train text transformer.
-transformer:
-	python3 transformer.py
diff --git a/models/README.md b/models/README.md
index 577f0c6..ed505bd 100644
--- a/models/README.md
+++ b/models/README.md
@@ -7,66 +7,33 @@ repositories (SR).
 
 You can use these models for detecting SRs:
 
-* [Random-Forest model](#random-forest-model)
-* [Isolation forest model](#isolation-forest-model)
-* [Transformer model](#transformer-model)
+* [Isolation forest model](#if-model)
+* [Transformer model based on BERT](#transformer-bert-model)
 
-## Random-Forest model
-
-In order to use trained model with [Random-Forest] learning algorithm:
-
-```python
-from src.rf_model import RfModel
-
-rating = RfModel().predict("<input here>")
-print(rating) # probability that repository is a SR
-```
-
-## Isolation forest model
+## IF model
 
 TBD..
 
-## Transformer model
+## Transformer BERT model
 
-You can use this transformer model for detecting SRs too:
-
-```python
-from src.transformer_model import TransformerModel
-
-rating = TransformerModel().predict("<input here>")
-print(rating) # probability that repository is a SR
-```
+TBD..
 
 ## How to train it?
 
-To train Random-Forest model, run this script:
+To train models, run this:
 
 ```bash
-make random-forest
+docker run -e "GH_TOKEN=..." abialiauski/samples-filter-models model/<model-name>.py
 ```
 
-After training successfully ended, you should have `.joblib` files for both,
-vectorizer and model. Latest model version is [here](https://github.com/h1alexbel/samples-filter/tree/random-forest).
-To publish a new, trigger [random-forest.yml](https://github.com/h1alexbel/samples-filter/actions/workflows/random-forest.yml)
-workflow.
+For `<model-name>` you should provide a name of Python script for training, for
+instance `isolation-forest` or `t_bert`. For `GH_TOKEN` you should provide a
+[GitHub PAT] for pushing `isolation-forest` model files into `results` branch
+of this `h1alexbel/samples-filter`. If you are training `t_bert` model, and you
+want to export output model files, then pass `-e "HF_TOKEN=..."` to push them
+to the [HuggingFace].
 
-If you want to train transformer model, you should run this script:
-
-```bash
-export HF_TOKEN=<your hugging face API key>
-make transformer
-```
-
-Pay attention to the exported `HF_TOKEN`, it's needed for [pushing](https://huggingface.co/docs/transformers/v4.15.0/en/model_sharing)
-trained model into Hugging Face Model Hub.
-
-Training will take approximately 10 minutes. After it successfully finished,
-all output model files will be pushed to [h1alexbel/github-samples-tclassifier](https://huggingface.co/h1alexbel/github-samples-tclassifier).
-
-To do it remotely, trigger [transformer.yml](https://github.com/h1alexbel/samples-filter/actions/workflows/transformer.yml)
-workflow.
-
-You will need [Python 3.9+] installed.
+You will need [Docker] installed.
 
 ## How to build new dataset?
 
@@ -74,8 +41,21 @@ Dataset used for model training are located here:
 [train.csv](https://github.com/h1alexbel/samples-filter/blob/dataset/train.csv)
 To refresh it, run [srdataset] either on cloud VM or locally. The building
 process can take a while. After it completed, you should have `dataset.csv`
-file with all collected repositories.
-
-[Random-Forest]: https://en.wikipedia.org/wiki/Random_forest
-[Python 3.9+]: https://www.python.org/downloads/release/python-390
+file with all collected repositories with the following structure:
+
+* `name`: repository full name, e.g. `redisson/redisson-examples`.
+* `readme`: repository README.md file.
+* `description`: repository description.
+* `topics`: a set of repository topics, e.g. `[apache, streaming, kafka]`
+* `CPD`: commits per day calculated metric.
+* `RC`: published releases to commits ratio.
+* `IC`: issues to commits ratio.
+
+All features must be preprocessed and vectorized using [pipeline.py].
+Once you have vectors, you can [feed](#how-to-train-it) them to the models.
+
+[GitHub PAT]: https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens
+[HuggingFace]: https://huggingface.co/models
+[Docker]: https://docs.docker.com/get-docker
 [srdataset]: https://github.com/h1alexbel/srdataset
+[pipeline.py]: https://github.com/h1alexbel/samples-filter/blob/master/models/model/pre/pipeline.py
diff --git a/models/model/isolation-forest.py b/models/model/isolation-forest.py
new file mode 100644
index 0000000..293620e
--- /dev/null
+++ b/models/model/isolation-forest.py
@@ -0,0 +1,28 @@
+# The MIT License (MIT)
+#
+# Copyright (c) 2024 Aliaksei Bialiauski
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included
+# in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
+# Train model with isolation forest algorithm.
+# @todo #129:90min Train isolation forest model.
+#  We need to find out how to properly train isolation forest model with
+#  gathered data in order to detect anomalies (SRs). Don't forget to remove
+#  this puzzle.
+print("training IF model...")
diff --git a/models/model/embeddings.py b/models/model/pre/embeddings.py
similarity index 86%
rename from models/model/embeddings.py
rename to models/model/pre/embeddings.py
index 8b0d289..49a4cfd 100644
--- a/models/model/embeddings.py
+++ b/models/model/pre/embeddings.py
@@ -39,6 +39,11 @@ def embed(self):
         print(f"Encoder: {self.tokenizer}, {self.model}, output length: {self.length}")
         inputs = []
         masks = []
+        # @todo #143:30min We generate embeddings for each token instead of the whole unit.
+        #   For now, we generate embeddings for each token. We probably should
+        #   generate embeddings for joined tokens as one unit. In this case we
+        #   can try to replace preprocessing steps with a huggingface tokenizers.
+        #   Let's validate this assumption.
         for tokens in self.tokens:
             ids = self.tokenizer.encode_plus(
                 tokens,
diff --git a/models/model/pipeline.py b/models/model/pre/pipeline.py
similarity index 89%
rename from models/model/pipeline.py
rename to models/model/pre/pipeline.py
index 01aa974..c7c3c10 100644
--- a/models/model/pipeline.py
+++ b/models/model/pre/pipeline.py
@@ -20,12 +20,12 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 # SOFTWARE.
 
-from model.embeddings import Embeddings
-from model.pre_description import PreDescription
-from model.pre_name import PreName
-from model.pre_readme import PreReadme
-from model.pre_topics import PreTopics
-from model.vector import Vector
+from model.pre.embeddings import Embeddings
+from model.pre.pre_description import PreDescription
+from model.pre.pre_name import PreName
+from model.pre.pre_readme import PreReadme
+from model.pre.pre_topics import PreTopics
+from model.pre.vector import Vector
 
 """
 Pipeline.
diff --git a/models/model/pre_description.py b/models/model/pre/pre_description.py
similarity index 100%
rename from models/model/pre_description.py
rename to models/model/pre/pre_description.py
diff --git a/models/model/pre_name.py b/models/model/pre/pre_name.py
similarity index 100%
rename from models/model/pre_name.py
rename to models/model/pre/pre_name.py
diff --git a/models/model/pre_readme.py b/models/model/pre/pre_readme.py
similarity index 100%
rename from models/model/pre_readme.py
rename to models/model/pre/pre_readme.py
diff --git a/models/model/pre_topics.py b/models/model/pre/pre_topics.py
similarity index 100%
rename from models/model/pre_topics.py
rename to models/model/pre/pre_topics.py
diff --git a/models/model/vector.py b/models/model/pre/vector.py
similarity index 100%
rename from models/model/vector.py
rename to models/model/pre/vector.py
diff --git a/models/model/t_bert.py b/models/model/t_bert.py
new file mode 100644
index 0000000..c613ae0
--- /dev/null
+++ b/models/model/t_bert.py
@@ -0,0 +1,28 @@
+# The MIT License (MIT)
+#
+# Copyright (c) 2024 Aliaksei Bialiauski
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included
+# in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
+# Train model based on BERT.
+# @todo #129:90min Train BERT transformer model.
+#  We need to find out how to properly train BERT model with
+#  gathered data in order to detect anomalies (SRs). Don't forget to remove
+#  this puzzle.
+print("training BERT based model...")


Test patch:
diff --git a/models/model_tests/test_embeddings.py b/models/model_tests/pre/test_embeddings.py
similarity index 97%
rename from models/model_tests/test_embeddings.py
rename to models/model_tests/pre/test_embeddings.py
index 547c216..f9fc378 100644
--- a/models/model_tests/test_embeddings.py
+++ b/models/model_tests/pre/test_embeddings.py
@@ -21,7 +21,7 @@
 # SOFTWARE.
 import unittest
 
-from model.embeddings import Embeddings
+from model.pre.embeddings import Embeddings
 
 """
 Test cases for Embeddings. 
diff --git a/models/model_tests/test_pipeline.py b/models/model_tests/pre/test_pipeline.py
similarity index 98%
rename from models/model_tests/test_pipeline.py
rename to models/model_tests/pre/test_pipeline.py
index cb3fd52..b005f9a 100644
--- a/models/model_tests/test_pipeline.py
+++ b/models/model_tests/pre/test_pipeline.py
@@ -21,7 +21,7 @@
 # SOFTWARE.
 import unittest
 
-from model.pipeline import Pipeline
+from model.pre.pipeline import Pipeline
 
 """
 Test cases for Pipeline.
diff --git a/models/model_tests/test_pre_description.py b/models/model_tests/pre/test_pre_description.py
similarity index 96%
rename from models/model_tests/test_pre_description.py
rename to models/model_tests/pre/test_pre_description.py
index d16d91f..2fbe04c 100644
--- a/models/model_tests/test_pre_description.py
+++ b/models/model_tests/pre/test_pre_description.py
@@ -21,7 +21,7 @@
 # SOFTWARE.
 import unittest
 
-from model.pre_description import PreDescription
+from model.pre.pre_description import PreDescription
 
 """
 Test cases for PreDescription.
diff --git a/models/model_tests/test_pre_name.py b/models/model_tests/pre/test_pre_name.py
similarity index 97%
rename from models/model_tests/test_pre_name.py
rename to models/model_tests/pre/test_pre_name.py
index 1534e1a..bcdf341 100644
--- a/models/model_tests/test_pre_name.py
+++ b/models/model_tests/pre/test_pre_name.py
@@ -21,7 +21,7 @@
 # SOFTWARE.
 import unittest
 
-from model.pre_name import PreName
+from model.pre.pre_name import PreName
 
 """
 Test cases for PreName.
diff --git a/models/model_tests/test_pre_readme.py b/models/model_tests/pre/test_pre_readme.py
similarity index 98%
rename from models/model_tests/test_pre_readme.py
rename to models/model_tests/pre/test_pre_readme.py
index ecf77ba..42df9bc 100644
--- a/models/model_tests/test_pre_readme.py
+++ b/models/model_tests/pre/test_pre_readme.py
@@ -21,7 +21,7 @@
 # SOFTWARE.
 import unittest
 
-from model.pre_readme import PreReadme
+from model.pre.pre_readme import PreReadme
 
 """
 Test cases for PreReadme.
diff --git a/models/model_tests/test_pre_topics.py b/models/model_tests/pre/test_pre_topics.py
similarity index 97%
rename from models/model_tests/test_pre_topics.py
rename to models/model_tests/pre/test_pre_topics.py
index f38bde4..71a5409 100644
--- a/models/model_tests/test_pre_topics.py
+++ b/models/model_tests/pre/test_pre_topics.py
@@ -21,7 +21,7 @@
 # SOFTWARE.
 import unittest
 
-from model.pre_topics import PreTopics
+from model.pre.pre_topics import PreTopics
 
 """
 Test cases for PreTopics.
diff --git a/models/model_tests/test_vector.py b/models/model_tests/pre/test_vector.py
similarity index 97%
rename from models/model_tests/test_vector.py
rename to models/model_tests/pre/test_vector.py
index 119894b..09a6946 100644
--- a/models/model_tests/test_vector.py
+++ b/models/model_tests/pre/test_vector.py
@@ -21,7 +21,7 @@
 # SOFTWARE.
 import unittest
 
-from model.vector import Vector
+from model.pre.vector import Vector
 
 """
 Test cases for Vector.