Repository: KnowledgeCaptureAndDiscovery/somef (language: python)

Issue:
Recognize installation instructions from available files
From file "installing.md" or "install.md".
This should be straightforward and can be found in many repos. Example:
https://github.com/facebookresearch/wav2letter/tree/v0.1
nc flag should be propagated to the category section too
Right now this category is always run


Discussion / hints:

Fix patch:
diff --git a/README.md b/README.md
index a48118fc..a9a5204f 100644
--- a/README.md
+++ b/README.md
@@ -37,19 +37,19 @@ Given a readme file (or a GitHub/Gitlab repository) SOMEF will extract the follo
 - **Releases**: Pointer to the available versions of a software component
 - **Changelog**: Description of the changes between versions
 - **Issue tracker**: Link where to open issues for the target repository
+- **Package distribution**: Links to package sites like pypi in case the repository has a package available.
 - **Programming languages**: Languages used in the repository
-- **Repository Status**: Repository status as it is described in [repostatus.org](https://www.repostatus.org/)
-- **Arxiv Links**: Links to Arxiv articles
+- **Repository Status**: Repository status as it is described in [repostatus.org](https://www.repostatus.org/).
 - **Stargazers count**: Total number of stargazers of the project
-- **Forks count**: Number of forks of the project
-- **Forks url**: Links to forks made of the project
+- **Forks count**: Number of forks of the project.
+- **Forks url**: Links to forks made of the project.
 - **Code of Conduct**: Link to the code of conduct of the project
 - **Scripts**: Snippets of code contained in the repository.
 - **Support channels**: Help channels one can use to get support about the target software component.
 - **Images**: Images used to illustrate the software component.
 - **Logo**: Main logo used to represent the target software component.
 - **Ontologies**: URL and path to the ontology files present in the repository.
-- **Category**: The category of the repository. Current supported categories include: Astrophisics, Audio, Computer vision, Graphs, Natural language processing, Reinforcement learning, Semantc web, Sequential. These categories have been extracted from [awesome lists](https://github.com/topics/awesome-list) and [Papers with code](https://paperswithcode.com/). Find more information in our [documentation](https://somef.readthedocs.io/en/latest/).
+- **Application domain**: The application domain of the repository. Current supported domains include: Astrophisics, Audio, Computer vision, Graphs, Natural language processing, Reinforcement learning, Semantc web, Sequential. Domains are not mutually exclusive. These domains have been extracted from [awesome lists](https://github.com/topics/awesome-list) and [Papers with code](https://paperswithcode.com/). Find more information in our [documentation](https://somef.readthedocs.io/en/latest/).
 
 
 We use different supervised classifiers, header analysis, regular expressions and the GitHub/Gitlab API to retrieve all these fields (more than one technique may be used for each field). Each extraction records its provenance, with the confidence and technique used on each step. For more information check the [output format description](https://somef.readthedocs.io/en/latest/output/)
@@ -240,10 +240,11 @@ Options:
                                   is easy to compare to another JSON output
                                   file.
 
-  -m, --missing                   JSON report with the missing metadata fields
-                                  SOMEF was not able to find. The report will
-                                  be placed in  $PATH_missing.json, where
-                                  $PATH is -o, -c or -g.
+  -m, --missing                   The JSON will include a field 
+                                  somef_missing_categories to report with the 
+                                  missing metadata fields that SOMEF was not 
+                                  able to find.
+
   -kt, --keep_tmp PATH            SOMEF will NOT delete the temporary folder
                                   where files are stored for analysis. Files
                                   will be stored at the
diff --git a/docs/index.md b/docs/index.md
index 47592322..7d6c5258 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -51,11 +51,11 @@ Given a readme file (or a GitHub repository) SOMEF will extract the following ca
 - **Images**: Images used to illustrate the software component.
 - **Logo**: Main logo used to represent the target software component.
 - **Ontologies**: URL and path to the ontology files present in the repository.
-- **Category**: The category of the repository <sup>[1](#myfootnote1)</sup> 
+- **Application domain**: The application domain of the repository. This may be related to the research area of a software component (e.g., Astrophysics) or the general domain/functionality of the tool (i.e., machine learning projects)<sup>[1](#myfootnote1)</sup> 
 
 SOMEF uses the [Software  Description Ontology](https://w3id.org/okn/o/sd), which extends [Schema.org](https://schema.org) and [Codemeta](https://codemeta.github.io/terms/) to represent all the categories listed above. To see different export options please see [the getting started page](https://somef.readthedocs.io/en/latest/usage/).
 
-<a name="myfootnote1">1</a> The available categories currently are the followings: 
+<a name="myfootnote1">1</a> The available application domains currently are: 
 
 - **Astrophisics**: a branch of space science that applies the laws of physics and chemistry to seek to understand the universe and our place in it. The field explores topics such as the birth, life and death of stars, planets, galaxies, nebulae and other objects in the universe.
 - **Audio**: a process of transforming, exploring, and interpreting audio signals recorded by digital devices.
diff --git a/docs/output.md b/docs/output.md
index a1909cf5..e870ec2b 100644
--- a/docs/output.md
+++ b/docs/output.md
@@ -1,29 +1,232 @@
 SOMEF supports three main output formats. Each of them contains different information with different levels of granularity. Below we enumerate them from more granular to less granular:
 
-### JSON format
-Simple JSON representation that indicates, for each extracted metadata category, the technique used for its extraction and its confidence, in addition to the detected excerpt. The JSON snippet below shows an example for the Description category of a Python library.
+## JSON format
+**Version:** 1.0.0
+
+Default SOMEF response (and more complete in terms of metadata). The JSON format returns a set of categories, as shown in the snippet below:
 
 ```json
-"description": [
+{
+  "<categoryName>": [
     {
-      "excerpt": "KGTK is a Python library ...",
-      "confidence": [0.8294290479925978],
-      "technique": "Supervised classification"
+      ...
     }
-  ]
+  ],
+  "<categoryName2>": [
+    {
+      ...
+    }
+  ],
+  "somef_provenance":{    
+    "date": "2022-05-20 12:00:00",
+    "somef_version": "0.9.1", 
+    "somef_schema_version":"1.0.0"
+  }
+}
+```
+In the snippet, each `<categoryName>` corresponds to the different categories SOMEF was able to find. An additional JSON field called `somef_provenance` returns provenance information of the SOMEF execution. The `somef_provenance` field always has the same two properties, as shown in the table below:
+
+| Property | Mandatory? | Expected value | Definition |
+|---|---|---|---|
+| **date** | Yes | Date | Date when the extraction was performed. Knowing the date is critical, as a repository may change its README file. |
+| **somef_version** | Yes | String | Version of SOMEF used to extract metadata from a code repository. |
+| **somef_schema_version** | Yes | String | Version of SOMEF schema used to represent the JSON output format. |
+
+!!! info
+    If a property is `mandatory` then it will always be returned in the  output JSON.
+
+### Category
+Each extracted metadata category is returned as a list, which contains the number of results SOMEF found about that category when exploring a code repository. For example, the snippet below shows a repository with two descriptions (a short one extracted from the GitHub API and a longer one extracted from the README file). SOMEF aims to return both of them:
+
+```json
+"description": [
+  {
+    "result": {
+        "value": "KGTK is a Python library ...",
+        "type": "text"
+    },
+    "confidence": 0.8294290479925978,
+    "technique": "Supervised classification",
+    "source": "<url to readme file>"
+  },
+  {
+    "result": {
+        "value": "Python library for large KG manipulation",
+        "type": "text"
+    },
+    "confidence": 1,
+    "technique": "GitHub API"
+  }
+]
+```
+For each element of the list, SOMEF returns a `result` object, together with its `confidence` value, the `technique` used in the extraction and the `source` file where it's coming from (in case there is one). For example, in the snippet above, SOMEF extracted a description from the README file of the repository using supervised classification, and a short description using the GitHub API.
+
+The `confidence` depends on the `technique` used. In this case, the confidence is driven by the classifier which makes the prediction. For the GitHub API the confidence is higher, as it was a description added manually by the authors.
+
+SOMEF aims to recognize the following categories (in alphabetical order):
+
+- `application_domain`: The application domain of the repository. This may be related to the research area of a software component (e.g., Astrophysics) or the general domain/functionality of the tool (i.e., machine learning projects). See all current recognized application domains [here](https://somef.readthedocs.io/en/latest/#myfootnote1).
+- `acknowledgement`: Any text that the authors have prepared to acknnowledge the contribution from others, or project funding.
+- `contributors`: Contributors to a software component
+- `contributing guidelines`: Guidelines indicating how to contribute to a software component.
+- `citation`: Software citation (usually in `.bib` form) as the authors have stated in their readme file, or through a `CFF` file.
+- `code_of_conduct`: Link to the code of conduct file of the project
+- `code_repository`: Link to the source code (typically the repository where the readme can be found)
+- `contact`: Contact person responsible for maintaining a software component.
+- `date_created`: Date when the software component was created.
+- `date_updated`: Date when the software component was last updated (note that this will always be older than the date of the extraction).
+- `description`: A description of what the software component does.
+- `documentation`: Where to find additional documentation about a software component.
+- `download`: Download instructions included in the repository.
+- `download_url`: URL where to download the target software (typically the installer, package or a tarball to a stable version)
+- `executable_example`: Jupyter notebooks ready for execution (e.g., through myBinder)
+- `faq`: Frequently asked questions about a software component
+- `forks_count`: Number of forks of the project at the time of the extraction.
+- `forks_url`: Links to forks made of the project (GitHub only)
+- `full_name`: Name + owner (owner/name) (if available)
+- `full_title`: If the repository has a short name, we will attempt to extract the longer version of the repository name. For example, a repository may be called "Widoco", but the longer title is "Wizard for documenting ontologies".
+- `has_build_file`: Build file to create a Docker image for the target software
+- `has_executable_notebook`: Jupyter notebooks included in a repository. 
+- `has_script_file`: Snippets of code contained in the repository.
+- `identifier`: Identifiers detected within a repository (e.g., Digital Object Identifier).
+- `images`: Images used to illustrate the software component.
+- `installation`: A set of instructions that indicate how to install a target repository
+- `invocation`: Execution command(s) needed to run a scientific software component
+- `issue_tracker`: Link where to open issues for the target repository
+- `keywords`: set of terms used to commonly identify a software component
+- `license`: License and usage terms of a software component
+- `logo`: Main logo used to represent the target software component.
+- `name`: Name identifying a software component
+- `ontologies`: URL and path to the ontology files present in the repository.
+- `owner`: Name of the user or organization in charge of the repository
+- `package_distribution`: Link to official package repositories where the software can be downloaded from (e.g., `pypi`).
+- `programming_languages`: Languages used in the repository.
+- `readme_url`: URL to the main README file in the repository.
+- `related_documentation`: Pointers to documentation of related projects which may be needed when using the target repository.
+- `releases`: Pointer to the available versions of a software component.
+- `repository_status`: Repository status as it is described in [repostatus.org](https://www.repostatus.org/).
+- `requirements`: Pre-requisites and dependencies needed to execute a software component.
+- `run`: Running instructions of a software component. It may be wider than the `invocation` category, as it may include several steps and explanations.
+- `stargazers_count`: Total number of stargazers of the project.
+- `support`: Guidelines and links of where to obtain support for a software component.
+- `support_channels`: Help channels one can use to get support about the target software component.
+- `usage`: Usage examples and considerations of a code repository.
+
+The following table summarized the properties used to describe a `category`:
+
+| Property | Mandatory? | Expected value | Definition |
+|---|---|---|---|
+| **confidence** | Yes | Number | Value ranging from 0 (very low)  to 1 (very high) indicating the confidence of the program in the quality of the extraction. |
+| **result** | Yes | Result | Result obtained from the extraction in a code repository |
+| **source** | No | Url | URL of the source file used for the extraction. |
+| **technique** | Yes | String | Technique used for the extraction. One of the following list: Supervised classification, header analysis, regular expression, GitHub API, File exploration, Code parsing |
+
+### Result
+Field returning the extracted output from the code repository. An example can be seen below for a citation found in BibteX format in a README file of a code repository:
+
+```json
+"citation": [
+  {
+    "result": {
+      "value": "@inproceedings{ilievski2020kgtk,\n  title={{KGTK}: A Toolkit for Large Knowledge Graph Manipulation and Analysis}},\n  author={Ilievski, Filip and Garijo, Daniel and Chalupsky, Hans and Divvala, Naren Teja and Yao, Yixiang and Rogers, Craig and Li, Ronpeng and Liu, Jun and Singh, Amandeep and Schwabe, Daniel and Szekely, Pedro},\n  booktitle={International Semantic Web Conference},\n  pages={278--293},\n  year={2020},\n  organization={Springer}\n  url={https://arxiv.org/pdf/2006.00088.pdf}\n}",
+      "format": "bibtex",
+      "type": "string",
+      "url": "https://arxiv.org/pdf/2006.00088.pdf"
+    },
+    "confidence": 1.0,
+    "technique": "Regular expression",
+    "source": "<url to README file>"
+  }
+]
 ```
-The `confidence` depends on the `technique` used. In this case, the confidence is driven by the classifier which makes the prediction. 
 
-The techniques can be of several types: `header analysis`, `supervised classification`, `file exploration`, `GitHub API` and `regular expression`. Among these, only `supervised classification` provides a confidence different to `1`.
+A result may have the following fields:
+
+| Property | Mandatory? | Expected value | Definition |
+|---|---|---|---|
+| **format** | No | String | Format in which the value is returned. For example, it may be a Dockerfile, a jupyter notebook, or a citation in BibteX.
+| **type** | Yes | String | Text indicating the value type of the result. In some cases it refers to the type of literal being returned, while in others it refers to the type of the object. For example, a  `license` may be detected as a URL, as a text excerpt detected from a file, or as an object with both name and url. |
+| **value** | Yes | String, Number, Date or Url | Text with the result of the extraction performed by SOMEF. The value is always a single object, not a list. |
+
+
+Depending on the `type` of the result, additional properties may be found. 
+
+The following object `types` are currently supported:
+
+- `Release`: software releases of the current code repository, as available from GitHub.
+- `Programming_language`: Programming language used in the repository. 
+- `License`: object representing all the metadata SOMEF extracts from a license.
+- `Agent`: user (typically, a person) or organization responsible for authoring a software release or a paper.
+- `Publication`: Scientific paper associated with the code repository.
+
+The following literal types are currently supported:
+
+- `Number`: A numerical value. We do not distinguish between integer, long or float.
+- `Date`: Dates in xsd:date format.
+- `String`: Any representation in text that is not considered a number, date or url. There are two special types of strings.
+  - `Text_excerpt`: The value is a string that has been extracted from a file.
+  - `File_dump`: The  value is a string with the contents of a file (e.g., a `citation.cff` file, or a `license.md` file).
+- `Url`: uniform resource locator of a file.
+
+The table below summarizes all types and their corresponding properties:
+
+| Property | Describes | Expected value | Definition |
+|---|---|---|---|
+| **author** | Release, Publication | Agent,  Organization | Person or organization responsible for creating an article or a software release. |
+| **doi** | Publication | Url | When a publication is detected, but the format is in bibtek or CFF, SOMEF will add a `doi`  field with the detected DOI value. The result includes a full URL. |
+| **description** | Release | String | Descriptive text with the purpose of the release |
+| **date_created** | Release | Date | Date of creation of a release |
+| **date_published** | Release | Date | Date of publication of a release |
+| **html_url** | Release | Url | link to the HTML representation of a release |
+| **name** | License, Release,  User, Publication, Programming_language | String | Title or name used to designate the release, license user, programming language or publication. |
+| **original_header** | Text_excerpt | String | If the result value is extracted from a markdown file like a README, the original header of that section is also returned. |
+| **parent_header** | Text_excerpt | [String] | If the result value is extracted from a markdown file like a README, the parent header(s) of the current section are also returned (in case they exist). |
+| **size** | Programming_language | Number | File size content (bytes) of a code repository using a given programming language |
+| **spdx_id** | License | String | Spdx id corresponding to this license |
+| **tag** | Release | String | named version of a release |
+| **tarball_url** | Release | Url | URL to the tar ball file where to download a software release |
+| **url** | Release, Publication, License | Url | Uniform resource locator of the resource |
+| **zipball_url** | Release | Url | URL to the zip file where to download a software release |
+
+
+### Format
+The following formats for a result value are currently recognized:
+
+- `bibtex`: format typically used to [document bibliography](https://www.bibtex.com/g/bibtex-format/) in LateX projects.
+- `cff`: [Citation file format](https://citation-file-format.github.io/), an increasingly popular format for citing software projects.
+- `jupyter_notebook`: [computational notebooks](https://ipython.org/ipython-doc/3/notebook/nbformat.html) typically used in data science.
+- `dockerfile`: [Docker files](https://docs.docker.com/engine/reference/builder/) used to build Docker images.
+- `docker_compose`: [orchestration file](https://docs.docker.com/compose/compose-file/) used to communicate multiple containers.
+- `readthedocs`: documentation format used by many repositories in order to describe their projects.
+- `wiki`: documentation format used in GitHub repositories.
+
+### Technique
+The techniques can be of several types: 
+
+- `header_analysis`: the result was extracted by analyzing the headers used in the README file and assessing their proximity to commonly used headers (and other synonims).
+- `supervised_classification`: the results were obtained after running text classifiers trained for detecting that type of header.
+- `file_exploration`: the result comes from an exploration of the files in the repository
+- `GitHub_API`: the result was obtained from the GitHub API.
+- `GitLab_API`: the result was obtained from the GitLab API.
+- `regular_expression`: the result was obtained after performing regular expressions on the files in the repository.
+- `code_parser`: the result was obtained from code configuration files with metadata markup. 
+
+### Missing categories
+If SOMEF is run with the `-m` flag, a report of the categories that the program was not able to find is returned. The format for this field is slightly different than the rest, providing a list of the missing categories. An example can be seen below:
+
+```json
+"somef_missing_categories": [
+  "description", 
+  "citation"
+]
+```
+In this case, SOMEF was not able to find a description or a citation in the target repository. Missing categories will not be added in Codemeta and Turtle exports. Note that the prefix `somef` is added in the field, to indicate that this is a special type of category.
 
-The output may also contain the following fields:
 
-- `format`: format in which a citation is recognized. Supported formats are `bibtex` and `citation file format`.
-- `doi`: Digital Object Identifier associated with a paper or a software repository.
-- `type`: type of documentation recognized by somef. There are two main recognized types: `readthedocs` and `wiki`.
+## Turtle format 
+RDF represents information in triples (subject, predicate and object), where the subject is the entity to be described (in this case a code repository), the predicate is the property describing the subject (e.g., `description`) and the object is the value obtained (which can be an object or a literal). Representing the provenance information returned by SOMEF is therefore challenging, and would require adopting reification mechanisms that make the output more complex. Therefore we simplify the output by providing our best guess for each of the extracted fields.
 
-### Turtle format
-RDF representation using the [Software Description Ontology](https://w3id.org/okn/o/sd/). The snippet below shows a sample description of a software entry. The `excerpt` and `confidence` fields are ommitted in this representation (every category with confidence above the threshold specified when running SOMEF will be included in the results)
+Our RDF representation uses the [Software Description Ontology](https://w3id.org/okn/o/sd/). The `excerpt` and `confidence` fields are ommitted in this representation (every category with confidence above the threshold specified when running SOMEF will be included in the results). 
 
-### Codemeta format
+## Codemeta format 
 JSON-LD representation following the [Codemeta specification](https://codemeta.github.io/) (which itself extends [Schema.org](https://schema.org/)). The `excerpt` and `confidence` fields are ommitted in this representation (every category with confidence above the threshold specified when running SOMEF will be included in the results). In addition, any metadata category outside from what is defined in Codemeta will be avoided.
diff --git a/setup.py b/setup.py
index 8d423024..9ac197e5 100644
--- a/setup.py
+++ b/setup.py
@@ -24,7 +24,8 @@
     "inflect>=5.4.0",
     "contractions>=0.1.66",
     "chardet==5.0.0",
-    "imbalanced-learn>=0.8.1"
+    "imbalanced-learn>=0.8.1",
+    "pytest"
 ]
 
 
@@ -70,7 +71,7 @@ def find_paths(dirname):
         "Intended Audience :: Science/Research",
         "Operating System :: Unix",
     ],
-    entry_points={"console_scripts": ["somef = somef.__main__:main"]},
+    entry_points={"console_scripts": ["somef = somef.__main__:cli"]},
     package_dir={"": "src"},
     packages=find_packages(where="src", exclude=["somef.tests*"]),
     package_data={"somef": find_package_data("src/somef")},
diff --git a/src/somef/__main__.py b/src/somef/__main__.py
index c5c4d887..34d7bcb4 100644
--- a/src/somef/__main__.py
+++ b/src/somef/__main__.py
@@ -142,7 +142,7 @@ def configure(auto, base_uri):
     "-m",
     is_flag=True,
     default=False,
-    help="""The JSON will include a category missingCategories to report with the missing metadata fields SOMEF was not 
+    help="""The JSON will include a field missing_categories to report with the missing metadata fields SOMEF was not 
     able to find. """
 )
 @click.option(
@@ -154,4 +154,4 @@ def configure(auto, base_uri):
 )
 def describe(**kwargs):
     somef_cli.run_cli(**kwargs)
-    click.secho(f"Success", fg="green")
+    click.secho(f"Success", fg="green")
\ No newline at end of file
diff --git a/src/somef/export/json_export.py b/src/somef/export/json_export.py
index ed735e26..7ef978fa 100644
--- a/src/somef/export/json_export.py
+++ b/src/somef/export/json_export.py
@@ -2,15 +2,28 @@
 from dateutil import parser as date_parser
 from ..utils import constants
 from .data_to_graph import DataGraph
-
-
-def save_json_output(repo_data, outfile, missing, pretty=False):
-    """Function that saves the final json Object in the output file"""
-    print("Saving json data to", outfile)
+from ..process_results import Result
+
+
+def save_json_output(repo_data, out_path, missing, pretty=False):
+    """
+    Function that saves the final json Object in the output file
+    Parameters
+    ----------
+    @param repo_data: dictionary with the metadata to be saved
+    @param out_path: output path where to save the JSON
+    @param missing: print the categories SOMEF was not able to find
+    @param pretty: format option to print the JSON in a human-readable way
+
+    Returns
+    -------
+    @return: Does not return a value
+    """
+    print("Saving json data to", out_path)
     if missing:
-        missing = create_missing_fields(repo_data)
-        repo_data["missingCategories"] = missing["missingCategories"]
-    with open(outfile, 'w') as output:
+        # add a new key-value papir to the dictionary
+        repo_data[constants.CAT_MISSING] = create_missing_fields(repo_data)
+    with open(out_path, 'w') as output:
         if pretty:
             json.dump(repo_data, output, sort_keys=True, indent=2)
         else:
@@ -18,133 +31,154 @@ def save_json_output(repo_data, outfile, missing, pretty=False):
 
 
 def save_codemeta_output(repo_data, outfile, pretty=False):
-    """Function that saves a JSONLD file with the codemeta results"""
+    """
+    Function that saves a Codemeta JSONLD file with a summary of the results
 
-    def data_path(path):
-        return DataGraph.resolve_path(repo_data, path)
+    Parameters
+    ----------
+    @param repo_data: JSON with the results to translate to Codemeta
+    @param outfile: path where to save the codemeta file
+    @param pretty: option to show the JSON results in a nice format
+    """
 
     def format_date(date_string):
         date_object = date_parser.parse(date_string)
         return date_object.strftime("%Y-%m-%d")
 
-    latest_release = None
-    releases = data_path(["releases", "excerpt"])
-
-    if releases is not None and len(releases) > 0:
-        latest_release = releases[0]
-        latest_pub_date = date_parser.parse(latest_release["datePublished"])
-        for index in range(1, len(releases)):
-            release = releases[index]
-            pub_date = date_parser.parse(release["datePublished"])
-
-            if pub_date > latest_pub_date:
-                latest_release = release
-                latest_pub_date = pub_date
-
-    def release_path(path):
-        return DataGraph.resolve_path(latest_release, path)
+    # latest_release = None
+    # releases = data_path(["releases", "excerpt"])
+    #
+    # if releases is not None and len(releases) > 0:
+    #     latest_release = releases[0]
+    #     latest_pub_date = date_parser.parse(latest_release["datePublished"])
+    #     for index in range(1, len(releases)):
+    #         release = releases[index]
+    #         pub_date = date_parser.parse(release["datePublished"])
+    #
+    #         if pub_date > latest_pub_date:
+    #             latest_release = release
+    #             latest_pub_date = pub_date
+
+    # def release_path(path):
+    #     return DataGraph.resolve_path(latest_release, path)
 
     code_repository = None
-    if "codeRepository" in repo_data:
-        code_repository = data_path(["codeRepository", "excerpt"])
-
-    author_name = data_path(["owner", "excerpt"])
+    if constants.CAT_CODE_REPOSITORY in repo_data:
+        code_repository = repo_data[constants.CAT_CODE_REPOSITORY][0][constants.PROP_RESULT][constants.PROP_VALUE]
 
-    # do the descriptions
+    author_name = None
+    if constants.CAT_OWNER in repo_data:
+        author_name = repo_data[constants.CAT_OWNER][0][constants.PROP_RESULT][constants.PROP_VALUE]
 
-    def average_confidence(x):
-        confs = x["confidence"]
-
-        if len(confs) > 0:
-            try:
-                return max(sum(confs) / len(confs))
-            except:
-                return 0
-        else:
-            return 0
-
-    descriptions = data_path(["description"])
+    descriptions = repo_data[constants.CAT_DESCRIPTION]
     descriptions_text = []
     if descriptions is not None:
-        descriptions.sort(key=lambda x: (average_confidence(x) + (1 if x["technique"] == "GitHub API" else 0)),
+        descriptions.sort(key=lambda x: (x[constants.PROP_CONFIDENCE] + (1 if x[constants.PROP_TECHNIQUE] == constants.GITHUB_API else 0)),
                           reverse=True)
-        descriptions_text = [x["excerpt"] for x in descriptions]
-
-    published_date = ""
-    try:
-        published_date = format_date(release_path(["datePublished"]))
-    except:
-        print("Published date is not available")
+        descriptions_text = [x[constants.PROP_RESULT][constants.PROP_VALUE] for x in descriptions]
 
     codemeta_output = {
         "@context": "https://doi.org/10.5063/schema/codemeta-2.0",
         "@type": "SoftwareSourceCode"
     }
-    if "license" in repo_data:
-        codemeta_output["license"] = data_path(["license", "excerpt"])
+    if constants.CAT_LICENSE in repo_data:
+        # We mix the name of the license from github API with the URL of the file (if found)
+        l_result = {}
+        for l in repo_data[constants.CAT_LICENSE]:
+            if constants.PROP_NAME in l[constants.PROP_RESULT].keys():
+                l_result["name"] = l[constants.PROP_RESULT][constants.PROP_NAME]
+            if "url" not in l_result.keys() and constants.PROP_URL in l[constants.PROP_RESULT].keys():
+                    l_result["url"] = l[constants.PROP_RESULT][constants.PROP_URL]
+            # We get the first license we find from the repo
+            elif l[constants.PROP_TECHNIQUE] == constants.TECHNIQUE_FILE_EXPLORATION \
+                    and constants.PROP_SOURCE in l.keys() and "api.github.com" in l_result["url"]:
+                l_result["url"] = l[constants.PROP_SOURCE]
+        codemeta_output["license"] = l_result
     if code_repository is not None:
         codemeta_output["codeRepository"] = code_repository
         codemeta_output["issueTracker"] = code_repository + "/issues"
-    if "dateCreated" in repo_data:
-        codemeta_output["dateCreated"] = format_date(data_path(["dateCreated", "excerpt"]))
-    if "dateModified" in repo_data:
-        codemeta_output["dateModified"] = format_date(data_path(["dateModified", "excerpt"]))
-    if "downloadUrl" in repo_data:
-        codemeta_output["downloadUrl"] = data_path(["downloadUrl", "excerpt"])
-    if "name" in repo_data:
-        codemeta_output["name"] = data_path(["name", "excerpt"])
-    if "logo" in repo_data:
-        codemeta_output["logo"] = data_path(["logo", "excerpt"])
-    if "releases" in repo_data:
-        codemeta_output["releaseNotes"] = release_path(["body"])
-        codemeta_output["version"] = release_path(["tag_name"])
-    if "topics" in repo_data:
-        codemeta_output["keywords"] = data_path(["topics", "excerpt"])
-    if "languages" in repo_data:
-        codemeta_output["programmingLanguage"] = data_path(["languages", "excerpt"])
-    if "requirement" in repo_data:
-        codemeta_output["softwareRequirements"] = data_path(["requirement", "excerpt"])
-    if "installation" in repo_data:
-        codemeta_output["buildInstructions"] = data_path(["installation", "excerpt"])
-    if "owner" in repo_data:
+    if constants.CAT_DATE_CREATED in repo_data:
+        codemeta_output["dateCreated"] = format_date(repo_data[constants.CAT_DATE_CREATED][0][constants.PROP_RESULT][constants.PROP_VALUE])
+    if constants.CAT_DATE_UPDATED in repo_data:
+        codemeta_output["dateModified"] = format_date(repo_data[constants.CAT_DATE_UPDATED][0][constants.PROP_RESULT][constants.PROP_VALUE])
+    if constants.CAT_DOWNLOAD_URL in repo_data:
+        codemeta_output["downloadUrl"] = repo_data[constants.CAT_DOWNLOAD_URL][0][constants.PROP_RESULT][constants.PROP_VALUE]
+    if constants.CAT_NAME in repo_data:
+        codemeta_output["name"] = repo_data[constants.CAT_NAME][0][constants.PROP_RESULT][constants.PROP_VALUE]
+    if constants.CAT_LOGO in repo_data:
+        codemeta_output["logo"] = repo_data[constants.CAT_LOGO][0][constants.PROP_RESULT][constants.PROP_VALUE]
+    if constants.CAT_KEYWORDS in repo_data:
+        codemeta_output["keywords"] = repo_data[constants.CAT_KEYWORDS][0][constants.PROP_RESULT][constants.PROP_VALUE]
+    if constants.CAT_PROGRAMMING_LANGUAGES in repo_data:
+        codemeta_output["programmingLanguage"] = [x[constants.PROP_RESULT][constants.PROP_VALUE] for x in repo_data[constants.CAT_PROGRAMMING_LANGUAGES]]
+    if constants.CAT_REQUIREMENTS in repo_data:
+        codemeta_output["softwareRequirements"] = [x[constants.PROP_RESULT][constants.PROP_VALUE] for x in repo_data[constants.CAT_REQUIREMENTS]]
+    install_links = []
+    if constants.CAT_INSTALLATION in repo_data:
+        for inst in repo_data[constants.CAT_INSTALLATION]:
+            if inst[constants.PROP_TECHNIQUE] == constants.TECHNIQUE_HEADER_ANALYSIS and constants.PROP_SOURCE in inst.keys():
+                install_links.append(inst[constants.PROP_SOURCE])
+            elif inst[constants.PROP_TECHNIQUE] == constants.TECHNIQUE_FILE_EXPLORATION:
+                install_links.append(inst[constants.PROP_RESULT][constants.PROP_VALUE])
+    if constants.CAT_DOCUMENTATION in repo_data:
+        for inst in repo_data[constants.CAT_DOCUMENTATION]:
+            if inst[constants.PROP_TECHNIQUE] == constants.TECHNIQUE_HEADER_ANALYSIS and constants.PROP_SOURCE in inst.keys():
+                install_links.append(inst[constants.PROP_SOURCE])
+            elif inst[constants.PROP_TECHNIQUE] == constants.TECHNIQUE_FILE_EXPLORATION or \
+                    inst[constants.PROP_TECHNIQUE] == constants.TECHNIQUE_REGULAR_EXPRESSION:
+                install_links.append(inst[constants.PROP_RESULT][constants.PROP_VALUE])
+    if len(install_links) > 0:
+        # remove duplicates and generate codemeta
+        install_links = list(set(install_links))
+        codemeta_output["buildInstructions"] = install_links
+    if constants.CAT_OWNER in repo_data:
+        # if user then person, otherwise organization
+        type_aux = repo_data[constants.CAT_OWNER][0][constants.PROP_RESULT][constants.PROP_TYPE]
+        if type_aux == "User":
+            type_aux = "Person"
         codemeta_output["author"] = [
             {
-                "@type": "Person",
+                "@type": type_aux,
                 "@id": "https://github.com/" + author_name
             }
         ]
-    if "citation" in repo_data:
-        codemeta_output["citation"] = data_path(["citation", "excerpt"])
-    if "identifier" in repo_data:
-        codemeta_output["identifier"] = data_path(["identifier", "excerpt"])
-    if "issueTracker" in repo_data:
-        codemeta_output["issueTracker"] = data_path(["issueTracker", "excerpt"])
-    if "readme_url" in repo_data:
-        codemeta_output["readme"] = data_path(["readme_url", "excerpt"])
-    if "contributors" in repo_data:
-        codemeta_output["contributor"] = data_path(["contributors", "excerpt"])
+    if constants.CAT_CITATION in repo_data:
+        url_cit = []
+        for cit in repo_data[constants.CAT_CITATION]:
+            if constants.PROP_DOI in cit[constants.PROP_RESULT].keys():
+                url_cit.append(cit[constants.PROP_RESULT][constants.PROP_DOI])
+            elif constants.PROP_FORMAT in cit[constants.PROP_RESULT].keys() \
+                    and cit[constants.PROP_RESULT][constants.PROP_FORMAT] == constants.FORMAT_CFF:
+                url_cit.append(cit[constants.PROP_SOURCE])
+        if len(url_cit) > 0:
+            codemeta_output["citation"] = url_cit
+    if constants.CAT_IDENTIFIER in repo_data:
+        codemeta_output["identifier"] = repo_data[constants.CAT_IDENTIFIER][0][constants.PROP_RESULT][constants.PROP_VALUE]
+    if constants.CAT_README_URL in repo_data:
+        codemeta_output["readme"] = repo_data[constants.CAT_README_URL][0][constants.PROP_RESULT][constants.PROP_VALUE]
+    # if "contributors" in repo_data:
+    #     codemeta_output["contributor"] = data_path(["contributors", "excerpt"])
+    # A person is expected, and we extract text at the moment
     if descriptions_text:
         codemeta_output["description"] = descriptions_text
-    if published_date != "":
-        codemeta_output["datePublished"] = published_date
+    # if published_date != "":
+    # commenting this out because we cannot assume the last published date to be the published date
+    #     codemeta_output["datePublished"] = published_date
     pruned_output = {}
 
     for key, value in codemeta_output.items():
         if not (value is None or ((isinstance(value, list) or isinstance(value, tuple)) and len(value) == 0)):
             pruned_output[key] = value
-
     # now, prune out the variables that are None
-
     save_json_output(pruned_output, outfile, None, pretty=pretty)
 
 
-def create_missing_fields(repo_data):
+def create_missing_fields(result):
     """Function to create a small report with the categories SOMEF was not able to find.
-    The categories are added to the JSON results. This won't be added if you only export TTL or Codemeta"""
+    The categories are added to the JSON results. This won't be added if you export TTL or Codemeta"""
     missing = []
-    out = {}
+    repo_data = result
     for c in constants.categories_files_header:
         if c not in repo_data:
             missing.append(c)
-    out["missingCategories"] = missing
-    return out
+    return missing
diff --git a/src/somef/extract_ontologies.py b/src/somef/extract_ontologies.py
index 1fa39cd4..1a3ca850 100644
--- a/src/somef/extract_ontologies.py
+++ b/src/somef/extract_ontologies.py
@@ -8,10 +8,11 @@ def is_file_ontology(file_path):
     This method is in a separate file in case we want to extract additional metadata if required
     Parameters
     ----------
-    file_path path of the candidate ontology
+    @param file_path: path of the candidate ontology
 
-    Returns the URI of the target ontology (if there is one)
+    Returns
     -------
+    @return: The URI of the target ontology (if there is one)
     """
     # load in rdf lib
     try:
@@ -28,6 +29,6 @@ def is_file_ontology(file_path):
         for r in g.query(q1):
             # print("Found that %s is an ontology" % file_path)
             return r.onto
-    except:
+    except Exception:
         # If the candidate file could not be read, pass
         pass
diff --git a/src/somef/header_analysis.py b/src/somef/header_analysis.py
index da3d0746..aaf4f61f 100644
--- a/src/somef/header_analysis.py
+++ b/src/somef/header_analysis.py
@@ -1,6 +1,4 @@
-## Main function: header_analysis(text)
-## input file: readme files text data
-## output file: json files with categories extracted using header analysis; other text data cannot be extracted
+import logging
 import re
 import string
 
@@ -8,7 +6,9 @@
 import pandas as pd
 from textblob import Word
 
+from .process_results import Result
 from .parser import mardown_parser
+from .utils import constants
 
 pd.options.mode.chained_assignment = None  # default='warn'
 
@@ -17,28 +17,28 @@
 
 # Word("citation").synsets[2] -> Includes ack, which is not the right sense
 citation = [Word("citation").synsets[3], Word("reference").synsets[1], Word("cite").synsets[3]]
-group.update({"citation": citation})
+group.update({constants.CAT_CITATION: citation})
 
 ack = [Word("acknowledgement").synsets[0]]
-group.update({"acknowledgement": ack})
+group.update({constants.CAT_ACKNOWLEDGEMENT: ack})
 
 run = [Word("run").synsets[9], Word("run").synsets[34], Word("execute").synsets[4]]
-group.update({"run": run})
+group.update({constants.CAT_RUN: run})
 
 install = [Word("installation").synsets[0], Word("install").synsets[0], Word("setup").synsets[1],
            Word("prepare").synsets[0], Word("preparation").synsets[0], Word("manual").synsets[0],
            Word("guide").synsets[2], Word("guide").synsets[9]]
-group.update({"installation": install})
+group.update({constants.CAT_INSTALLATION: install})
 
 download = [Word("download").synsets[0]]
-group.update({"download": download})
+group.update({constants.CAT_DOWNLOAD: download})
 
 requirement = [Word("requirement").synsets[2], Word("prerequisite").synsets[0], Word("prerequisite").synsets[1],
                Word("dependency").synsets[0], Word("dependent").synsets[0]]
-group.update({"requirement": requirement})
+group.update({constants.CAT_REQUIREMENTS: requirement})
 
 contact = [Word("contact").synsets[9]]
-group.update({"contact": contact})
+group.update({constants.CAT_CONTACT: contact})
 
 description = [Word("description").synsets[0], Word("description").synsets[1],
                Word("introduction").synsets[3], Word("introduction").synsets[6],
@@ -46,16 +46,19 @@
                Word("initiation").synsets[1],
                #               Word("overview").synsets[0],
                Word("summary").synsets[0], Word("summary").synsets[2]]
-group.update({"description": description})
+group.update({constants.CAT_DESCRIPTION: description})
 
 contributor = [Word("contributor").synsets[0]]
-group.update({"contributor": contributor})
+group.update({constants.CAT_CONTRIBUTORS: contributor})
+
+contributing = [Word("contributing").synsets[1]]
+group.update({constants.CAT_CONTRIBUTING_GUIDELINES: contributing})
 
 documentation = [Word("documentation").synsets[1]]
-group.update({"documentation": documentation})
+group.update({constants.CAT_DOCUMENTATION: documentation})
 
 license = [Word("license").synsets[3], Word("license").synsets[0]]
-group.update({"license": license})
+group.update({constants.CAT_LICENSE: license})
 
 usage = [Word("usage").synsets[0], Word("example").synsets[0], Word("example").synsets[5],
          # Word("implement").synsets[1],Word("implementation").synsets[1],
@@ -63,7 +66,7 @@
          Word("tutorial").synsets[1],
          Word("start").synsets[0], Word("start").synsets[4], Word("started").synsets[0],
          Word("started").synsets[1], Word("started").synsets[7], Word("started").synsets[8]]
-group.update({"usage": usage})
+group.update({constants.CAT_USAGE: usage})
 
 # update = [Word("updating").synsets[0], Word("updating").synsets[3]]
 # group.update({"update": update})
@@ -72,27 +75,26 @@
 # Word("issues").synsets[0],
 faq = [Word("errors").synsets[5], Word("problems").synsets[0],
        Word("problems").synsets[2], Word("faq").synsets[0]]
-group.update({"faq": faq})
+group.update({constants.CAT_FAQ: faq})
 
 support = [Word("support").synsets[7], Word("help").synsets[0], Word("help").synsets[9], Word("report").synsets[0],
            Word("report").synsets[6]]
-group.update({"support": support})
+group.update({constants.CAT_SUPPORT: support})
 
 
 def extract_bash_code(text):
     """Function to detect code blocks"""
-    splitted = text.split("```")
+    split = text.split("```")
     output = []
-    if (len(splitted) >= 3):
-        for index, value in enumerate(splitted):
+    if len(split) >= 3:
+        for index, value in enumerate(split):
             if index % 2 == 1:
-                output.append(splitted[index])
+                output.append(split[index])
     return output
 
 
 def extract_header_content(text):
     """Function designed to extract headers and contents of text and place it in a dataframe"""
-    print('Extracting headers and content.')
     header = []
     headers = mardown_parser.extract_headers(text)
     for key in headers.keys():
@@ -188,64 +190,75 @@ def clean_html(text):
     return cleantext
 
 
-def extract_categories_using_headers(text):
-    """Main function to extract categories using headers"""
-    data, none_header_content = extract_header_content(text)
-    print('Labeling headers.')
-    if data.empty:
-        print("File to analyze has no headers")
-        return {}, [text]
-    data['Group'] = data['Header'].apply(lambda row: label_header(row))
-    data['GroupParent'] = data['ParentHeader'].apply(lambda row: label_parent_headers(row))
-    for i in data.index:
-        if len(data['Group'][i]) == 0 and len(data['GroupParent'][i]) > 0:
-            data.at[i, 'Group'] = data['GroupParent'][i]
-    data = data.drop(columns=['GroupParent'])
-    if len(data['Group'].iloc[0]) == 0:
-        data['Group'].iloc[0] = ['unknown']
-    groups = data.apply(lambda x: pd.Series(x['Group']), axis=1).stack().reset_index(level=1, drop=True)
-
-    groups.name = 'Group'
-    data = data.drop('Group', axis=1).join(groups)
-    if data['Group'].iloc[0] == 'unknown':
-        data['Group'].iloc[0] = np.NaN
-
-    # to json
-    group = data.loc[(data['Group'] != 'None') & pd.notna(data['Group'])]
-    # group = group.reindex(columns=['Content', 'Group'])
-    group['confidence'] = [[1.0]] * len(group)
-    group.rename(columns={'Content': 'excerpt'}, inplace=True)
-    group.rename(columns={'Header': 'originalHeader'}, inplace=True)
-    group.rename(columns={'ParentHeader': 'parentHeader'}, inplace=True)
-    group['technique'] = 'Header extraction'
-    # group['original header'] = 'NaN'
-    group_json = group.groupby('Group').apply(lambda x: x.to_dict('r')).to_dict()
-    for key in group_json.keys():
-        for ind in range(len(group_json[key])):
-            del group_json[key][ind]['Group']
-
-    # for key in group_json.keys():
-    #     for ind in range(len(group_json[key])):
-    #         print(group_json[key][ind]['excerpt'])
-
-    print('Converting to json files.')
-
-    # strings without tag (they will be classified)
-    str_list = data.loc[data['Group'].isna(), ['Content']].values.squeeze().tolist()
-    if type(str_list) != list:
-        str_list = [str_list]
-
-    if none_header_content != None:
-        str_list.append(none_header_content)
-
-    # remove empty field parentHeader
-    for key in group_json.keys():
-        elements = group_json[key]
-        new_elements = []
-        for element in elements:
-            if element['parentHeader'] == "":
-                del element['parentHeader']
-            new_elements.append(element)
-        group_json[key] = new_elements
-
-    return group_json, str_list
+def extract_categories(repo_data, repository_metadata: Result):
+    """
+    Function that adds category information extracted using header information
+    Parameters
+    ----------
+    @param repo_data: data to use the header analysis
+    @param repository_metadata: Result object with the results found so far in the repo
+
+    Returns
+    -------
+    @return Result with the information added.
+    """
+    logging.info("Extracting information using headers")
+    if repo_data is None or repo_data == "" or len(repo_data) == 0:
+        return repository_metadata, []
+    try:
+        data, none_header_content = extract_header_content(repo_data)
+        logging.info('Labeling headers.')
+        if data.empty:
+            logging.warning("File to analyze has no headers")
+            return repository_metadata, [repo_data]
+        data['Group'] = data['Header'].apply(lambda row: label_header(row))
+        data['GroupParent'] = data['ParentHeader'].apply(lambda row: label_parent_headers(row))
+        for i in data.index:
+            if len(data['Group'][i]) == 0 and len(data['GroupParent'][i]) > 0:
+                data.at[i, 'Group'] = data['GroupParent'][i]
+        data = data.drop(columns=['GroupParent'])
+        if len(data['Group'].iloc[0]) == 0:
+            data['Group'].iloc[0] = ['unknown']
+        groups = data.apply(lambda x: pd.Series(x['Group']), axis=1).stack().reset_index(level=1, drop=True)
+
+        groups.name = 'Group'
+        data = data.drop('Group', axis=1).join(groups)
+        if data['Group'].iloc[0] == 'unknown':
+            data['Group'].iloc[0] = np.NaN
+
+        # to json
+        group = data.loc[(data['Group'] != 'None') & pd.notna(data['Group'])]
+        group.rename(columns={'Content': constants.PROP_VALUE}, inplace=True)
+        group.rename(columns={'Header': constants.PROP_ORIGINAL_HEADER}, inplace=True)
+        group.rename(columns={'ParentHeader': constants.PROP_PARENT_HEADER}, inplace=True)
+        for index, row in group.iterrows():
+            source = ""
+            if constants.CAT_README_URL in repository_metadata.results.keys():
+                source = repository_metadata.results[constants.CAT_README_URL][0]
+                source = source[constants.PROP_RESULT][constants.PROP_VALUE]
+            parent_header = ""
+            if row[constants.PROP_PARENT_HEADER] != "":
+                parent_header = row.loc[constants.PROP_PARENT_HEADER]
+            result = {
+                constants.PROP_VALUE: row.loc[constants.PROP_VALUE],
+                constants.PROP_TYPE: constants.TEXT_EXCERPT,
+                constants.PROP_ORIGINAL_HEADER: row.loc[constants.PROP_ORIGINAL_HEADER]
+            }
+            if parent_header != "" and len(parent_header) > 0:
+                result[constants.PROP_PARENT_HEADER] = parent_header
+            if source != "":
+                repository_metadata.add_result(row.Group, result, 1, constants.TECHNIQUE_HEADER_ANALYSIS, source)
+            else:
+                repository_metadata.add_result(row.Group, result, 1, constants.TECHNIQUE_HEADER_ANALYSIS)
+
+        # strings without tag (they will be classified)
+        string_list = data.loc[data['Group'].isna(), ['Content']].values.squeeze().tolist()
+        if type(string_list) != list:
+            string_list = [string_list]
+        if none_header_content is not None and none_header_content != "":
+            string_list.append(none_header_content.strip())
+        logging.info("Header information extracted.")
+        return repository_metadata, string_list
+    except Exception as e:
+        logging.error("Error while extracting headers: ", str(e))
+        return repository_metadata, [repo_data]
diff --git a/src/somef/parser/create_excerpts.py b/src/somef/parser/create_excerpts.py
index 45b6bc49..14faada2 100644
--- a/src/somef/parser/create_excerpts.py
+++ b/src/somef/parser/create_excerpts.py
@@ -1,6 +1,6 @@
 from io import StringIO
 
-
+import logging
 from markdown import Markdown
 from ..utils import markdown_utils
 from . import mardown_parser
@@ -57,11 +57,10 @@ def create_excerpts(string_list):
     -------
     Extracted excerpts
     """
-    print("Splitting text into valid excerpts for classification")
+    logging.info("Splitting text into valid excerpts for classification")
     string_list = markdown_utils.remove_bibtex(string_list)
-    # divisions = createExcerpts.split_into_excerpts(string_list)
     divisions = mardown_parser.extract_blocks_excerpts(string_list)
-    print("Text Successfully split. \n")
+    logging.info("Text Successfully split. \n")
     output = {}
     for division in divisions:
         original = division
diff --git a/src/somef/parser/mardown_parser.py b/src/somef/parser/mardown_parser.py
index 856073e0..1f920921 100644
--- a/src/somef/parser/mardown_parser.py
+++ b/src/somef/parser/mardown_parser.py
@@ -1,3 +1,5 @@
+import logging
+
 import markdown
 import re
 import pandas as pd
@@ -56,12 +58,11 @@ def extract_content_per_header(original_text, headers):
     if len(text_tokenized) == 1:
         return text_tokenized[0]
     top_index = get_position(0, text_tokenized, top)
-    # si hay algo antes de la primera cabecera, no se procesa porque no está relacionado con ninguna cabecera
+    # do not process the text above all other headers, as it does not belong to one
     if top_index > 0:
         none_header_content = get_text(0, top_index, text_tokenized)
         none_header_content = regular_expressions.remove_links_images(none_header_content)
         none_header_content = regular_expressions.remove_html_tags(none_header_content)
-        #print(none_header_content)
     offset = 1
     if not text_tokenized[top_index].startswith('#'):
         offset = 2
@@ -91,6 +92,9 @@ def get_position(init_index, text_tokenized, text):
     while init_index < len(text_tokenized):
         val = text_tokenized[init_index]
         val = remove_hash(val).strip()
+        text = markdown.markdown(text).replace('<p>', '').replace('</p>', '').strip()
+        # Since we read the headers with HTML there could be encoding conversions line ;amp that could be lost
+        val = markdown.markdown(val).replace('<p>', '').replace('</p>', '').strip()
         if val.startswith(text):
             return init_index
         init_index += 1
@@ -265,8 +269,6 @@ def extract_text_excerpts_header(original_text):
                         header_content = header_content[1:]
                     content.append(header_content)
                     output[top] = header_content
-                # else:
-                    # print('No se procesa-->' + top)
                 top = bottom
                 top_index = bottom_index
 
@@ -430,7 +432,7 @@ def update_parents(new_header, parents):
             else:
                 return parent, parent_list
     except ValueError:
-        print("Error when retrieving the parent header list")
+        logging.error("Error when retrieving the parent header list")
 
     return parent, parent_list
 
diff --git a/src/somef/process_files.py b/src/somef/process_files.py
index 7c9510c3..f67fc79f 100644
--- a/src/somef/process_files.py
+++ b/src/somef/process_files.py
@@ -2,19 +2,21 @@
 import os
 import re
 import urllib
-from .utils import constants ,markdown_utils
+from .utils import constants, markdown_utils
 from . import extract_ontologies
+from .process_results import Result
 from chardet import detect
 
 
-def process_repository_files(repo_dir, filtered_resp, repo_type, owner="", repo_name="", repo_default_branch=""):
+def process_repository_files(repo_dir, metadata_result: Result, repo_type, owner="", repo_name="",
+                             repo_default_branch=""):
     """
     Method that given a folder, it recognizes whether there are notebooks, dockerfiles, docs, script files or
     ontologies.
     Parameters
     ----------
     @param repo_dir: path to the dir to analyze
-    @param filtered_resp: JSON object to be completed by this method
+    @param metadata_result: JSON object gathering the metadata found by SOMEF
     @param repo_type: GITHUB, GITLAB or LOCAL
     @param owner: owner of the repo (only for github/gitlab repos)
     @param repo_name: repository name (only for github/gitlab repos)
@@ -24,11 +26,6 @@ def process_repository_files(repo_dir, filtered_resp, repo_type, owner="", repo_
     -------
     @return: text of the main readme and a JSON dictionary (filtered_resp) with the findings in files
     """
-    notebooks = []
-    dockerfiles = []
-    docs = []
-    script_files = []
-    ontologies = []
     text = ""
     try:
         for dir_path, dir_names, filenames in os.walk(repo_dir):
@@ -36,21 +33,31 @@ def process_repository_files(repo_dir, filtered_resp, repo_type, owner="", repo_
             for filename in filenames:
                 file_path = os.path.join(repo_relative_path, filename)
                 if filename == "Dockerfile" or filename.lower() == "docker-compose.yml":
-                    if repo_type == constants.RepositoryType.GITHUB:
-                        dockerfiles.append(convert_to_raw_user_content_github(file_path, owner, repo_name, repo_default_branch))
-                    elif repo_type == constants.RepositoryType.GITLAB:
-                        dockerfiles.append(convert_to_raw_user_content_gitlab(file_path, owner, repo_name, repo_default_branch))
+                    docker_url = get_file_link(repo_type, file_path, owner, repo_name, repo_default_branch, repo_dir,
+                                               repo_relative_path, filename)
+                    if filename == "Dockerfile":
+                        format_file = constants.FORMAT_DOCKERFILE
                     else:
-                        dockerfiles.append(os.path.join(repo_dir, file_path))
-
+                        format_file = constants.FORMAT_DOCKER_COMPOSE
+                    metadata_result.add_result(constants.CAT_HAS_BUILD_FILE,
+                                               {
+                                                   constants.PROP_VALUE: docker_url,
+                                                   constants.PROP_TYPE: constants.URL,
+                                                   constants.PROP_FORMAT: format_file
+                                               },
+                                               1,
+                                               constants.TECHNIQUE_FILE_EXPLORATION, docker_url)
                 if filename.lower().endswith(".ipynb"):
-                    if repo_type == constants.RepositoryType.GITHUB:
-                        notebooks.append(convert_to_raw_user_content_github(file_path, owner, repo_name, repo_default_branch))
-                    elif repo_type == constants.RepositoryType.GITLAB:
-                        notebooks.append(convert_to_raw_user_content_gitlab(file_path, owner, repo_name, repo_default_branch))
-                    else:
-                        notebooks.append(os.path.join(repo_dir, file_path))
-
+                    notebook_url = get_file_link(repo_type, file_path, owner, repo_name, repo_default_branch, repo_dir,
+                                                 repo_relative_path, filename)
+                    metadata_result.add_result(constants.CAT_EXECUTABLE_EXAMPLE,
+                                               {
+                                                   constants.PROP_VALUE: notebook_url,
+                                                   constants.PROP_TYPE: constants.URL,
+                                                   constants.PROP_FORMAT: constants.FORMAT_JUPYTER_NB
+                                               },
+                                               1,
+                                               constants.TECHNIQUE_FILE_EXPLORATION, notebook_url)
                 filename_no_ext = os.path.splitext(filename)[0]
                 # this will take into account README, README.MD, README.TXT, README.RST
                 if "README" == filename_no_ext.upper():
@@ -61,137 +68,90 @@ def process_repository_files(repo_dir, filtered_resp, repo_type, owner="", repo_
                                 try:
                                     text = data_file_text.decode("utf-8")
                                 except UnicodeError as err:
-                                    print(f"{type(err).__name__} was raised: {err} Trying other encodings...")
+                                    logging.error(f"{type(err).__name__} was raised: {err} Trying other encodings...")
                                     text = data_file_text.decode(detect(data_file_text)["encoding"])
                                 if repo_type == constants.RepositoryType.GITHUB:
-                                    filtered_resp['readmeUrl'] = convert_to_raw_user_content_github(filename, owner,
-                                                                                                    repo_name,
-                                                                                                    repo_default_branch)
+                                    readme_url = convert_to_raw_user_content_github(filename, owner,
+                                                                                    repo_name,
+                                                                                    repo_default_branch)
+                                    metadata_result.add_result(constants.CAT_README_URL,
+                                                               {
+                                                                   constants.PROP_VALUE: readme_url,
+                                                                   constants.PROP_TYPE: constants.URL
+                                                               },
+                                                               1,
+                                                               constants.TECHNIQUE_FILE_EXPLORATION)
                         except ValueError:
-                            print("README Error: error while reading file content")
-                            print(f"{type(err).__name__} was raised: {err}")
-
-                if "LICENSE" == filename.upper() or "LICENSE.MD" == filename.upper():
-                    try:
-                        with open(os.path.join(dir_path, filename), "rb") as data_file:
-                            file_text = data_file.read()
-                            filtered_resp["licenseText"] = markdown_utils.unmark(file_text)
-                    except:
-                        # TO DO: try different encodings
-                        if repo_type == constants.RepositoryType.GITHUB:
-                            filtered_resp["licenseFile"] = convert_to_raw_user_content_github(filename, owner, repo_name,
-                                                                                              repo_default_branch)
-                        elif repo_type == constants.RepositoryType.GITLAB:
-                            filtered_resp["licenseFile"] = convert_to_raw_user_content_gitlab(filename, owner, repo_name,
-                                                                                              repo_default_branch)
-                        else:
-                            filtered_resp["licenseFile"] = os.path.join(repo_dir, repo_relative_path, filename)
-
+                            logging.error("README Error: error while reading file content")
+                            logging.error(f"{type(err).__name__} was raised: {err}")
+                if "LICENCE" == filename.upper() or "LICENSE" == filename.upper() or "LICENSE.MD" == filename.upper():
+                    # to do (issue 530) if there are two licenses, keep the one closer to the root
+                    metadata_result = get_file_content_or_link(repo_type, file_path, owner, repo_name,
+                                                               repo_default_branch,
+                                                               repo_dir, repo_relative_path, filename, dir_path,
+                                                               metadata_result, constants.CAT_LICENSE)
                 if "CODE_OF_CONDUCT" == filename.upper() or "CODE_OF_CONDUCT.MD" == filename.upper():
-                    if repo_type == constants.RepositoryType.GITHUB:
-                        filtered_resp["codeOfConduct"] = convert_to_raw_user_content_github(filename, owner, repo_name,
-                                                                                            repo_default_branch)
-                    elif repo_type == constants.RepositoryType.GITLAB:
-                        filtered_resp["codeOfConduct"] = convert_to_raw_user_content_gitlab(filename, owner, repo_name,
-                                                                                            repo_default_branch)
-                    else:
-                        filtered_resp["codeOfConduct"] = os.path.join(repo_dir, repo_relative_path, filename)
-
+                    metadata_result = get_file_content_or_link(repo_type, file_path, owner, repo_name,
+                                                               repo_default_branch,
+                                                               repo_dir, repo_relative_path, filename, dir_path,
+                                                               metadata_result, constants.CAT_COC)
                 if "CONTRIBUTING" == filename.upper() or "CONTRIBUTING.MD" == filename.upper():
-                    try:
-                        with open(os.path.join(dir_path, filename), "r") as data_file:
-                            file_text = data_file.read()
-                            filtered_resp["contributingGuidelines"] = markdown_utils.unmark(file_text)
-                    except:
-                        if repo_type == constants.RepositoryType.GITHUB:
-                            filtered_resp["contributingGuidelinesFile"] = convert_to_raw_user_content_github(filename,
-                                                                                                             owner,
-                                                                                                             repo_name,
-                                                                                                             repo_default_branch)
-                        elif repo_type == constants.RepositoryType.GITLAB:
-                            filtered_resp["contributingGuidelinesFile"] = convert_to_raw_user_content_gitlab(filename,
-                                                                                                             owner,
-                                                                                                             repo_name,
-                                                                                                             repo_default_branch)
-                        else:
-                            filtered_resp["contributingGuidelinesFile"] = os.path.join(repo_dir, repo_relative_path,
-                                                                                       filename)
+                    metadata_result = get_file_content_or_link(repo_type, file_path, owner, repo_name,
+                                                               repo_default_branch,
+                                                               repo_dir, repo_relative_path, filename, dir_path,
+                                                               metadata_result, constants.CAT_CONTRIBUTING_GUIDELINES)
+
                 if "ACKNOWLEDGMENT" in filename.upper() or "ACKNOWLEDGEMENT" in filename.upper():
-                    try:
-                        with open(os.path.join(dir_path, filename), "r") as data_file:
-                            file_text = data_file.read()
-                            filtered_resp["acknowledgement"] = markdown_utils.unmark(file_text)
-                    except ValueError:
-                        if repo_type == constants.RepositoryType.GITHUB:
-                            filtered_resp["acknowledgmentsFile"] = convert_to_raw_user_content_github(filename, owner,
-                                                                                                      repo_name,
-                                                                                                      repo_default_branch)
-                        elif repo_type == constants.RepositoryType.GITLAB:
-                            filtered_resp["acknowledgmentsFile"] = convert_to_raw_user_content_gitlab(filename, owner,
-                                                                                                      repo_name,
-                                                                                                      repo_default_branch)
-                        else:
-                            filtered_resp["acknowledgmentsFile"] = os.path.join(repo_dir, repo_relative_path, filename)
+                    metadata_result = get_file_content_or_link(repo_type, file_path, owner, repo_name,
+                                                               repo_default_branch,
+                                                               repo_dir, repo_relative_path, filename, dir_path,
+                                                               metadata_result, constants.CAT_ACKNOWLEDGEMENT)
                 if "CONTRIBUTORS" == filename.upper() or "CONTRIBUTORS.MD" == filename.upper():
-                    try:
-                        with open(os.path.join(dir_path, filename), "r") as data_file:
-                            file_text = data_file.read()
-                            filtered_resp["contributors"] = markdown_utils.unmark(file_text)
-                    except ValueError:
-                        if repo_type == constants.RepositoryType.GITHUB:
-                            filtered_resp["contributorsFile"] = convert_to_raw_user_content_github(filename, owner,
-                                                                                                   repo_name,
-                                                                                                   repo_default_branch)
-                        elif repo_type == constants.RepositoryType.GITLAB:
-                            filtered_resp["contributorsFile"] = convert_to_raw_user_content_gitlab(filename, owner,
-                                                                                                   repo_name,
-                                                                                                   repo_default_branch)
-                        else:
-                            filtered_resp["contributorsFile"] = os.path.join(repo_dir, repo_relative_path, filename)
-                if "CITATION" == filename.upper() or "CITATION.CFF" == filename.upper() or "CITATION.BIB" == filename.upper():
-                    try:
-                        with open(os.path.join(dir_path, filename), "r") as data_file:
-                            file_text = data_file.read()
-                            filtered_resp["citation"] = markdown_utils.unmark(file_text)
-                    except ValueError:
-                        if repo_type == constants.RepositoryType.GITHUB:
-                            filtered_resp["citationFile"] = convert_to_raw_user_content_github(filename, owner,
-                                                                                               repo_name,
-                                                                                               repo_default_branch)
-                        elif repo_type == constants.RepositoryType.GITLAB:
-                            filtered_resp["citationFile"] = convert_to_raw_user_content_gitlab(filename, owner,
-                                                                                               repo_name,
-                                                                                               repo_default_branch)
-                        else:
-                            filtered_resp["citationFile"] = os.path.join(repo_dir, repo_relative_path, filename)
-
+                    metadata_result = get_file_content_or_link(repo_type, file_path, owner, repo_name,
+                                                               repo_default_branch,
+                                                               repo_dir, repo_relative_path, filename, dir_path,
+                                                               metadata_result, constants.CAT_CONTRIBUTORS)
+                if "INSTALL" in filename.upper() and filename.upper().endswith("MD"):
+                    metadata_result = get_file_content_or_link(repo_type, file_path, owner, repo_name,
+                                                               repo_default_branch,
+                                                               repo_dir, repo_relative_path, filename, dir_path,
+                                                               metadata_result, constants.CAT_INSTALLATION)
+                # TO DO: double-check the formats and create a proper publication object (issue 207)
+                if "CITATION" == filename.upper() or "CITATION.BIB" == filename.upper():
+                    metadata_result = get_file_content_or_link(repo_type, file_path, owner, repo_name,
+                                                               repo_default_branch,
+                                                               repo_dir, repo_relative_path, filename, dir_path,
+                                                               metadata_result, constants.CAT_CITATION,
+                                                               constants.FORMAT_BIB)
+                if "CITATION.CFF" == filename.upper():
+                    metadata_result = get_file_content_or_link(repo_type, file_path, owner, repo_name,
+                                                               repo_default_branch,
+                                                               repo_dir, repo_relative_path, filename, dir_path,
+                                                               metadata_result, constants.CAT_CITATION,
+                                                               constants.FORMAT_CFF)
                 if filename.endswith(".sh"):
-                    if repo_type == constants.RepositoryType.GITHUB:
-                        script_files.append(convert_to_raw_user_content_github(file_path, owner, repo_name, repo_default_branch))
-                    elif repo_type == constants.RepositoryType.GITLAB:
-                        script_files.append(convert_to_raw_user_content_gitlab(file_path, owner, repo_name, repo_default_branch))
-                    else:
-                        script_files.append(os.path.join(repo_dir, file_path))
-
+                    sh_url = get_file_link(repo_type, file_path, owner, repo_name, repo_default_branch, repo_dir,
+                                           repo_relative_path, filename)
+                    metadata_result.add_result(constants.CAT_HAS_SCRIPT_FILE,
+                                               {
+                                                   constants.PROP_VALUE: sh_url,
+                                                   constants.PROP_TYPE: constants.URL
+                                               }, 1, constants.TECHNIQUE_FILE_EXPLORATION
+                                               )
                 if filename.endswith(".ttl") or filename.endswith(".owl") or filename.endswith(".nt") or filename. \
                         endswith(".xml"):
                     uri = extract_ontologies.is_file_ontology(os.path.join(repo_dir, file_path))
                     if uri is not None:
-                        # and not any(o['uri'] == uri for o in ontologies): This checks if the onto is not already
-                        # there, but we return all ontologies we find right now. Filtering is up to users
-                        file_url = ""
-                        if repo_type == constants.RepositoryType.GITHUB:
-                            file_url = convert_to_raw_user_content_github(file_path, owner, repo_name, repo_default_branch)
-                        elif repo_type == constants.RepositoryType.GITLAB:
-                            file_url = convert_to_raw_user_content_gitlab(file_path, owner, repo_name, repo_default_branch)
-                        else:
-                            file_url = os.path.join(repo_dir, file_path)
-                        onto = {
-                            "uri": uri,
-                            "file_url": file_url
-                        }
-                        ontologies.append(onto)
+                        onto_url = get_file_link(repo_type, file_path, owner, repo_name, repo_default_branch, repo_dir,
+                                                 repo_relative_path, filename)
+                        metadata_result.add_result(constants.CAT_ONTOLOGIES,
+                                                   {
+                                                       constants.PROP_VALUE: onto_url,
+                                                       constants.PROP_TYPE: constants.URL
+                                                   }, 1, constants.TECHNIQUE_FILE_EXPLORATION
+                                                   )
 
+            # TO DO: Improve this a bit, as just returning the docs folder is not that informative
             for dir_name in dir_names:
                 if dir_name.lower() == "docs":
                     if repo_relative_path == ".":
@@ -202,37 +162,101 @@ def process_repository_files(repo_dir, filtered_resp, repo_type, owner="", repo_
                             docs_path = os.path.join(new_repo_relative_path, dir_name)
                         else:
                             docs_path = os.path.join(repo_relative_path, dir_name)
-
                     names = os.listdir(os.path.join(repo_dir, docs_path))
                     for name in names:
                         if name.lower().endswith(".pdf") or name.lower().endswith(".md") or name.lower().endswith(
                                 ".html") or name.lower().endswith(".htm"):
                             if repo_type == constants.RepositoryType.GITHUB:
-                                docs.append(
-                                    f"https://github.com/{owner}/{repo_name}/tree/{urllib.parse.quote(repo_default_branch)}/{docs_path}")
+                                docs_url = f"https://github.com/{owner}/{repo_name}/tree/{urllib.parse.quote(repo_default_branch)}/{docs_path}"
                             elif repo_type == constants.RepositoryType.GITLAB:
-                                docs.append(
-                                    f"https://gitlab.com/{owner}/{repo_name}/-/tree/{urllib.parse.quote(repo_default_branch)}/{docs_path}")
+                                docs_url = f"https://gitlab.com/{owner}/{repo_name}/-/tree/{urllib.parse.quote(repo_default_branch)}/{docs_path}"
                             else:
-                                docs.append(os.path.join(repo_dir, docs_path))
+                                docs_url = os.path.join(repo_dir, docs_path)
+                            # docs.append(docs_url)
+                            metadata_result.add_result(constants.CAT_DOCUMENTATION,
+                                                       {
+                                                           constants.PROP_VALUE: docs_url,
+                                                           constants.PROP_TYPE: constants.URL
+                                                       }, 1, constants.TECHNIQUE_FILE_EXPLORATION
+                                                       )
                             break
-
-        if len(notebooks) > 0:
-            filtered_resp["hasExecutableNotebook"] = notebooks
-        if len(dockerfiles) > 0:
-            filtered_resp["hasBuildFile"] = dockerfiles
-        if len(docs) > 0:
-            filtered_resp["hasDocumentation"] = docs
-        if len(script_files) > 0:
-            filtered_resp["hasScriptFile"] = script_files
-        if len(ontologies) > 0:
-            filtered_resp["ontologies"] = ontologies
-        return text, filtered_resp
+        return text, metadata_result
     except TypeError:
         logging.error("Error when opening the repository files")
         return None, None
 
 
+def get_file_link(repo_type, file_path, owner, repo_name, repo_default_branch, repo_dir, repo_relative_path,
+                  filename):
+    """
+    Function to return the URL (or path if not possible) of a given file
+    Parameters
+    ----------
+    repo_type
+    file_path
+    owner
+    repo_name
+    repo_default_branch
+    repo_dir
+    repo_relative_path
+    filename
+
+    Returns
+    -------
+
+    """
+    if repo_type == constants.RepositoryType.GITHUB:
+        return convert_to_raw_user_content_github(file_path, owner, repo_name, repo_default_branch)
+    elif repo_type == constants.RepositoryType.GITLAB:
+        return convert_to_raw_user_content_gitlab(file_path, owner, repo_name, repo_default_branch)
+    else:
+        return os.path.join(repo_dir, repo_relative_path, filename)
+
+
+def get_file_content_or_link(repo_type, file_path, owner, repo_name, repo_default_branch, repo_dir, repo_relative_path,
+                             filename, dir_path, metadata_result: Result, category, format_result=""):
+    """
+    This method will return to the JSON file the contents of the file or its link if it cannot process the contents
+    Parameters
+    ----------
+    repo_type
+    file_path
+    owner
+    repo_name
+    repo_default_branch
+    repo_dir
+    repo_relative_path
+    filename
+    dir_path
+    metadata_result
+    category
+    format_result
+
+    Returns
+    -------
+    @returns String with the file content or url link to the file
+    """
+    url = get_file_link(repo_type, file_path, owner, repo_name, repo_default_branch, repo_dir, repo_relative_path,
+                        filename)
+    try:
+        with open(os.path.join(dir_path, filename), "r") as data_file:
+            file_text = data_file.read()
+            result = {
+                constants.PROP_VALUE: file_text,
+                constants.PROP_TYPE: constants.FILE_DUMP
+            }
+            if format_result != "":
+                result[constants.PROP_FORMAT] = format_result
+            metadata_result.add_result(category, result, 1, constants.TECHNIQUE_FILE_EXPLORATION, url)
+    except:
+        metadata_result.add_result(category,
+                                   {
+                                       constants.PROP_VALUE: url,
+                                       constants.PROP_TYPE: constants.URL
+                                   }, 1, constants.TECHNIQUE_FILE_EXPLORATION)
+    return metadata_result
+
+
 def convert_to_raw_user_content_github(partial, owner, repo_name, repo_ref):
     """Converts GitHub paths into raw.githubuser content URLs, accessible by users"""
     if partial.startswith("./"):
diff --git a/src/somef/process_repository.py b/src/somef/process_repository.py
index 4c240b45..fce61b11 100644
--- a/src/somef/process_repository.py
+++ b/src/somef/process_repository.py
@@ -9,6 +9,7 @@
 from urllib.parse import urlparse
 from .utils import constants
 from . import configuration
+from .process_results import Result
 
 
 # the same as requests.get(args).json(), but protects against rate limiting
@@ -24,12 +25,12 @@ def rate_limit_get(*args, backoff_rate=2, initial_backoff=1, **kwargs):
         rate_limit_remaining = data.headers["x-ratelimit-remaining"]
         epochtime = int(data.headers["x-ratelimit-reset"])
         date_reset = datetime.fromtimestamp(epochtime)
-        print("Remaining GitHub API requests: " + rate_limit_remaining + " ### Next rate limit reset at: " + str(
+        logging.info("Remaining GitHub API requests: " + rate_limit_remaining + " ### Next rate limit reset at: " + str(
             date_reset))
         response = response.json()
         if 'message' in response and 'API rate limit exceeded' in response['message']:
             rate_limited = True
-            print(f"rate limited. Backing off for {initial_backoff} seconds")
+            logging.warning(f"rate limited. Backing off for {initial_backoff} seconds")
             time.sleep(initial_backoff)
             # increase the backoff for next time
             initial_backoff *= backoff_rate
@@ -39,12 +40,13 @@ def rate_limit_get(*args, backoff_rate=2, initial_backoff=1, **kwargs):
     return response, date
 
 
-def load_gitlab_repository_metadata(repository_url):
+def load_gitlab_repository_metadata(repo_metadata: Result, repository_url):
     """
     Function uses the repository_url provided to load required information from gitlab.
     Information kept from the repository is written in keep_keys.
     Parameters
     ----------
+    @param repo_metadata: Result object with the metadata found in the repository so far
     @param repository_url: URL of the Gitlab repository to analyze
 
     Returns
@@ -63,7 +65,7 @@ def load_gitlab_repository_metadata(repository_url):
     path_components = url.path.split('/')
 
     if len(path_components) < 3:
-        print("Gitlab link is not correct.")
+        logging.error("Gitlab link is not correct.")
         return " ", {}
 
     owner = path_components[1]
@@ -73,7 +75,7 @@ def load_gitlab_repository_metadata(repository_url):
 
     project_id = get_project_id(repository_url)
     project_api_url = f"https://gitlab.com/api/v4/projects/{project_id}"
-    print(f"Downloading {project_api_url}")
+    logging.info(f"Downloading {project_api_url}")
     details = requests.get(project_api_url)
     project_details = details.json()
     date = details.headers["date"]
@@ -84,7 +86,7 @@ def load_gitlab_repository_metadata(repository_url):
 
     if len(path_components) >= 5:
         if not path_components[4] == "tree":
-            print(
+            logging.error(
                 "GitLab link is not correct. \nThe correct format is https://gitlab.com/{owner}/{repo_name}.")
 
             return " ", {}
@@ -93,70 +95,55 @@ def load_gitlab_repository_metadata(repository_url):
         default_branch = "/".join(path_components[5:])
         ref_param = {"ref": default_branch}
 
-    print(repo_api_base_url)
     if 'defaultBranch' in project_details.keys():
         general_resp = {'defaultBranch': project_details['defaultBranch']}
     elif 'default_branch' in project_details.keys():
         general_resp = {'defaultBranch': project_details['default_branch']}
+    else:
+        logging.error("Could not retrieve information for the GitLab repository")
+        return repo_metadata
 
     if 'message' in general_resp:
         if general_resp['message'] == "Not Found":
-            print("Error: repository name is incorrect")
+            logging.error("Error: repository name is incorrect")
         else:
             message = general_resp['message']
-            print("Error: " + message)
+            logging.error("Error: " + message)
 
         raise GithubUrlError
 
     if default_branch is None:
         default_branch = general_resp['defaultBranch']
 
-    # get only the fields that we want
-    def do_crosswalk(data, crosswalk_table):
-        def get_path(obj, path):
-            if isinstance(path, list) or isinstance(path, tuple):
-                if len(path) == 1:
-                    path = path[0]
-                else:
-                    return get_path(obj[path[0]], path[1:])
-
-            if obj is not None and path in obj:
-                return obj[path]
-            else:
-                return None
-
-        output = {}
-        for codemeta_key, path in crosswalk_table.items():
-            value = get_path(data, path)
-            if value is not None:
-                output[codemeta_key] = value
-            else:
-                print(f"Error: key {path} not present in gitlab repository")
-        return output
-
     # filtered_resp = do_crosswalk(general_resp, github_crosswalk_table)
-    filtered_resp = {"downloadUrl": f"https://gitlab.com/{owner}/{repo_name}/-/branches"}
+    # filtered_resp = {"downloadUrl": f"https://gitlab.com/{owner}/{repo_name}/-/branches"}
+    repo_metadata.add_result(constants.CAT_DOWNLOAD_URL,
+                             {
+                                 constants.PROP_VALUE: f"https://gitlab.com/{owner}/{repo_name}/-/branches",
+                                 constants.PROP_TYPE: constants.URL
+                             }, 1, constants.TECHNIQUE_GITLAB_API)
 
     # condense license information
-    license_info = {}
-    if 'license' in filtered_resp:
-        for k in ('name', 'url'):
-            if k in filtered_resp['license']:
-                license_info[k] = filtered_resp['license'][k]
+    license_result = {constants.PROP_TYPE: constants.URL}
+    if 'license' in general_resp:
+        if "name" in general_resp['license']:
+            license_result[constants.PROP_NAME] = general_resp["license"]["name"]
+        if "url" in general_resp['license']:
+            license_result[constants.PROP_VALUE] = general_resp["license"]["url"]
+        # for k in ('name', 'url'):
+        #     if k in general_resp['license']:
+        #         license_info[k] = general_resp['license'][k]
 
     # If we didn't find it, look for the license
-    if 'url' not in license_info or license_info['url'] is None:
-
+    if constants.PROP_VALUE not in license_result or license_result[constants.PROP_VALUE] is None:
         possible_license_url = f"{repository_url}/-/blob/master/LICENSE"
         license_text_resp = requests.get(possible_license_url)
-
-        # todo: It's possible that this request will get rate limited. Figure out how to detect that.
         if license_text_resp.status_code == 200:
             # license_text = license_text_resp.text
-            license_info['url'] = possible_license_url
+            license_result[constants.PROP_VALUE] = possible_license_url
 
-    if license_info != '':
-        filtered_resp['license'] = license_info
+    if constants.PROP_VALUE in license_result:
+        repo_metadata.add_result(constants.CAT_LICENSE, license_result, 1, constants.TECHNIQUE_GITLAB_API)
 
     # get keywords / topics
     # topics_headers = header
@@ -165,66 +152,58 @@ def get_path(obj, path):
     #                                   headers=topics_headers)
     topics_resp = {}
 
+    keywords = []
     if 'message' in topics_resp.keys():
-        print("Topics Error: " + topics_resp['message'])
+        logging.error("Topics Error: " + topics_resp['message'])
     elif topics_resp and 'names' in topics_resp.keys():
-        filtered_resp['topics'] = topics_resp['names']
+        keywords = topics_resp['names']
 
     if project_details['topics'] is not None:
-        filtered_resp['topics'] = project_details['topics']
+        keywords = project_details['topics']
+
+    if len(keywords) > 0:
+        value = '%s,' % (', '.join(keywords))
+        value = value.rstrip(',')
+        result = {
+            constants.PROP_VALUE: value,
+            constants.PROP_TYPE: constants.STRING
+        }
+        repo_metadata.add_result(constants.CAT_KEYWORDS, result, 1, constants.TECHNIQUE_GITLAB_API)
 
     # get social features: stargazers_count
-    stargazers_info = {}
-    # if 'stargazers_count' in filtered_resp:
     if project_details['star_count'] is not None:
-        stargazers_info['count'] = project_details['star_count']
-        stargazers_info['date'] = date
-        if 'stargazers_count' in filtered_resp.keys():
-            del filtered_resp['stargazers_count']
-    filtered_resp['stargazersCount'] = stargazers_info
+        result = {
+            constants.PROP_VALUE: project_details['star_count'],
+            constants.PROP_TYPE: constants.NUMBER
+        }
+        repo_metadata.add_result(constants.CAT_STARS, result, 1, constants.TECHNIQUE_GITLAB_API)
 
     # get social features: forks_count
-    forks_info = {}
-    # if 'forks_count' in filtered_resp:
     if project_details['forks_count'] is not None:
-        forks_info['count'] = project_details['forks_count']
-        forks_info['date'] = date
-        if 'forks_count' in filtered_resp.keys():
-            del filtered_resp['forks_count']
-    filtered_resp['forksCount'] = forks_info
+        repo_metadata.add_result(constants.CAT_FORK_COUNTS, {
+            constants.PROP_VALUE: project_details['forks_count'],
+            constants.PROP_TYPE: constants.NUMBER
+        }, 1, constants.TECHNIQUE_GITLAB_API)
+
+    # get programming languages
+    if 'languages_url' in project_details.keys():
+        if "message" in project_details['languages_url']:
+            logging.error("Languages Error: " + project_details['languages_url']["message"])
+        else:
+            result = {
+                constants.PROP_VALUE: list(project_details['languages_url']),
+                constants.PROP_TYPE: constants.STRING
+            }
+            repo_metadata.add_result(constants.CAT_PROGRAMMING_LANGUAGES, result, 1, constants.TECHNIQUE_GITLAB_API)
 
-    # get languages
-    languages = {}
-    filtered_resp['languages_url'] = "languages_url"
-    if "message" in languages:
-        print("Languages Error: " + languages["message"])
-    else:
-        filtered_resp['languages'] = list(languages.keys())
-
-    del filtered_resp['languages_url']
-
-    readme_info = {}
-    if 'message' in readme_info.keys():
-        print("README Error: " + readme_info['message'])
-        # text = ""
-    elif 'content' in readme_info:
-        readme = base64.b64decode(readme_info['content']).decode("utf-8")
-        # text = readme
-        filtered_resp['readmeUrl'] = readme_info['html_url']
-
-    if 'readme_url' in project_details:
-        # text = get_readme_content(project_details['readme_url'])
-        filtered_resp['readmeUrl'] = project_details['readme_url']
-
-    releases_list = {}
-    if isinstance(releases_list, dict) and 'message' in releases_list.keys():
-        print("Releases Error: " + releases_list['message'])
-    else:
-        filtered_resp['releases'] = [do_crosswalk(release, constants.release_crosswalk_table) for release in
-                                     releases_list]
+    if 'readme_url' in project_details.keys():
+        repo_metadata.add_result(constants.CAT_README_URL, {
+            constants.PROP_VALUE: project_details['readme_url'],
+            constants.PROP_TYPE: constants.URL
+        }, 1, constants.TECHNIQUE_GITLAB_API)
 
-    print("Repository information successfully loaded. \n")
-    return filtered_resp, owner, repo_name, default_branch
+    logging.info("Repository information successfully loaded. \n")
+    return repo_metadata, owner, repo_name, default_branch
 
 
 def download_gitlab_files(directory, owner, repo_name, repo_branch, repo_ref):
@@ -247,7 +226,7 @@ def download_gitlab_files(directory, owner, repo_name, repo_branch, repo_ref):
     repo_archive_url = f"https://gitlab.com/{owner}/{repo_name}/-/archive/{repo_branch}/{repo_name}-{repo_branch}.zip"
     if len(path_components) == 4:
         repo_archive_url = f"https://gitlab.com/{owner}/{repo_name}/-/archive/{repo_branch}/{path_components[3]}.zip"
-    print(f"Downloading {repo_archive_url}")
+    logging.info(f"Downloading {repo_archive_url}")
     repo_download = requests.get(repo_archive_url)
     repo_zip = repo_download.content
 
@@ -261,10 +240,12 @@ def download_gitlab_files(directory, owner, repo_name, repo_branch, repo_ref):
         zip_ref.extractall(repo_extract_dir)
 
     repo_folders = os.listdir(repo_extract_dir)
-    assert (len(repo_folders) == 1)
-
-    repo_dir = os.path.join(repo_extract_dir, repo_folders[0])
-    return repo_dir
+    if len(repo_folders) == 1:
+        repo_dir = os.path.join(repo_extract_dir, repo_folders[0])
+        return repo_dir
+    else:
+        logging.error("Error when processing GitLab repository")
+        return None
 
 
 def download_readme(owner, repo_name, default_branch, repo_type):
@@ -305,23 +286,24 @@ def download_readme(owner, repo_name, default_branch, repo_type):
     return text
 
 
-def load_online_repository_metadata(repository_url, ignore_api_metadata=False,
+def load_online_repository_metadata(repository_metadata: Result, repository_url, ignore_api_metadata=False,
                                     repo_type=constants.RepositoryType.GITHUB):
     """
     Function uses the repository_url provided to load required information from GitHub or Gitlab.
     Information kept from the repository is written in keep_keys.
     Parameters
     ----------
+    @param repository_metadata: Result object to store the findings by SOMEF. This is the returned object
     @param repo_type: type of the repository (GITLAB, GITHUB or LOCAL)
     @param ignore_api_metadata: true if you do not want to do an additional request to the target API
     @param repository_url: target repository URL.
 
     Returns
     -------
-    @return: Dictionary with the available metadata from online APIs, its owner, repo name and default branch
+    @return: Result object with the available metadata from online APIs plus its owner, repo name and default branch
     """
     if repo_type == constants.RepositoryType.GITLAB:
-        return load_gitlab_repository_metadata(repository_url)
+        return load_gitlab_repository_metadata(repository_metadata, repository_url)
     elif repo_type == constants.RepositoryType.LOCAL:
         logging.warning("Trying to download metadata from a local repository")
         return None
@@ -330,53 +312,50 @@ def load_online_repository_metadata(repository_url, ignore_api_metadata=False,
     # Read from the config file the right token information
     header = {}
     file_paths = configuration.get_configuration_file()
-    if 'Authorization' in file_paths.keys():
-        header['Authorization'] = file_paths['Authorization']
-    header['accept'] = 'application/vnd.github.v3+json'
+    if constants.CONF_AUTHORIZATION in file_paths.keys():
+        header[constants.CONF_AUTHORIZATION] = file_paths[constants.CONF_AUTHORIZATION]
+    header['accept'] = constants.GITHUB_ACCEPT_HEADER
 
     # load general response of the repository
     if repository_url[-1] == '/':
         repository_url = repository_url[:-1]
     url = urlparse(repository_url)
-    if url.netloc != 'github.com':
+    if url.netloc != constants.GITHUB_DOMAIN:
         logging.error("Repository must be from Github")
-        return " ", {}
+        return repository_metadata, "", "", ""
 
     path_components = url.path.split('/')
 
     if len(path_components) < 3:
         logging.error("Repository link is not correct. \nThe correct format is https://github.com/{owner}/{repo_name}.")
-        return " ", {}
+        return repository_metadata, "", "", ""
 
     owner = path_components[1]
     repo_name = path_components[2]
-
-    repo_api_base_url = f"https://api.github.com/repos/{owner}/{repo_name}"
-
+    repo_api_base_url = f"{constants.GITHUB_API}/{owner}/{repo_name}"
     default_branch = None
 
     if len(path_components) >= 5:
         if not path_components[3] == "tree":
-            print(
-                "Github link is not correct. \nThe correct format is https://github.com/{owner}/{repo_name}/tree/{ref}.")
-
-            return " ", {}
+            logging.error(
+                "Github link is not correct. \n"
+                "The correct format is https://github.com/{owner}/{repo_name}/tree/{ref}.")
+            return repository_metadata, "", "", ""
 
         # we must join all after 4, as sometimes tags have "/" in them.
         default_branch = "/".join(path_components[4:])
-        ref_param = {"ref": default_branch}
+        # ref_param = {"ref": default_branch}
 
     general_resp = {}
     date = ""
     if not ignore_api_metadata:
         general_resp, date = rate_limit_get(repo_api_base_url, headers=header)
-
     if 'message' in general_resp:
         if general_resp['message'] == "Not Found":
-            print("Error: Repository name is private or incorrect")
+            logging.error("Error: Repository name is private or incorrect")
         else:
             message = general_resp['message']
-            print("Error: " + message)
+            logging.error("Error: " + message)
 
         raise GithubUrlError
 
@@ -385,112 +364,116 @@ def load_online_repository_metadata(repository_url, ignore_api_metadata=False,
     elif default_branch is None:
         default_branch = general_resp['default_branch']
 
-    # get only the fields that we want
-    def do_crosswalk(data, crosswalk_table):
-        def get_path(obj, path):
-            if isinstance(path, list) or isinstance(path, tuple):
-                if len(path) == 1:
-                    path = path[0]
-                else:
-                    return get_path(obj[path[0]], path[1:])
-
-            if obj is not None and path in obj:
-                return obj[path]
-            else:
-                return None
-
-        output = {}
-        for codemeta_key, path in crosswalk_table.items():
-            value = get_path(data, path)
-            if value is not None:
-                output[codemeta_key] = value
-            else:
-                print(f"Error: key {path} not present in github repository")
-        return output
-
+    # filter the general response with only the fields we are interested in, mapping them to our keys
     filtered_resp = {}
     if not ignore_api_metadata:
         filtered_resp = do_crosswalk(general_resp, constants.github_crosswalk_table)
-        if "issueTracker" in filtered_resp:
-            issue_tracker = filtered_resp["issueTracker"]
-            issue_tracker = issue_tracker.replace("{/number}", "")
-            filtered_resp["issueTracker"] = issue_tracker
-
     # add download URL
-    filtered_resp["downloadUrl"] = f"https://github.com/{owner}/{repo_name}/releases"
-
-    # condense license information
-    license_info = {}
-    if 'license' in filtered_resp:
-        for k in ('name', 'url'):
-            if k in filtered_resp['license']:
-                license_info[k] = filtered_resp['license'][k]
-
-    # If we didn't find it, look for the license
-    if 'url' not in license_info or license_info['url'] is None:
-
-        possible_license_url = f"https://raw.githubusercontent.com/{owner}/{repo_name}/{default_branch}/LICENSE"
-        license_text_resp = requests.get(possible_license_url)
-
-        # todo: It's possible that this request will get rate limited. Figure out how to detect that.
-        if license_text_resp.status_code == 200:
-            # license_text = license_text_resp.text
-            license_info['url'] = possible_license_url
-
-    if len(license_info) > 0:
-        filtered_resp['license'] = license_info
-
-    # get keywords / topics
-    if 'topics' in general_resp.keys():
-        filtered_resp['topics'] = general_resp['topics']
-    # else:
-    #    topics_headers = header
-    #    topics_headers['accept'] = 'application/vnd.github.mercy-preview+json'
-    #    topics_resp, date = rate_limit_get(repo_api_base_url + "/topics",
-    #                                       headers=topics_headers)
-
-    #    if 'message' in topics_resp.keys():
-    #        print("Topics Error: " + topics_resp['message'])
-    #    elif topics_resp and 'names' in topics_resp.keys():
-    #        filtered_resp['topics'] = topics_resp['names']
-
-    # get social features: stargazers_count
-    stargazers_info = {}
-    if 'stargazers_count' in filtered_resp:
-        stargazers_info['count'] = filtered_resp['stargazers_count']
-        stargazers_info['date'] = date
-        del filtered_resp['stargazers_count']
-    if len(stargazers_info.keys()) > 0:
-        filtered_resp['stargazersCount'] = stargazers_info
-
-    # get social features: forks_count
-    forks_info = {}
-    if 'forks_count' in filtered_resp:
-        forks_info['count'] = filtered_resp['forks_count']
-        forks_info['date'] = date
-        del filtered_resp['forks_count']
-    if len(forks_info.keys()) > 0:
-        filtered_resp['forksCount'] = forks_info
-
+    filtered_resp[constants.CAT_DOWNLOAD_URL] = f"https://github.com/{owner}/{repo_name}/releases"
+
+    for category, value in filtered_resp.items():
+        value_type = constants.STRING
+        if category in constants.all_categories:
+            if category == constants.CAT_ISSUE_TRACKER:
+                value = value.replace("{/number}", "")
+            if category == constants.CAT_OWNER:
+                value_type = filtered_resp[constants.AGENT_TYPE]
+            if category == constants.CAT_KEYWORDS:
+                # we concatenate all keywords in a list, as the return value is always a single object
+                value = '%s,' % (', '.join(value))
+                value = value.rstrip(',')
+            if category in [constants.CAT_CODE_REPOSITORY, constants.CAT_ISSUE_TRACKER,
+                            constants.CAT_DOWNLOAD_URL, constants.CAT_FORKS_URLS]:
+                value_type = constants.URL
+            if category in [constants.CAT_DATE_CREATED, constants.CAT_DATE_UPDATED]:
+                value_type = constants.DATE
+            if category in [constants.CAT_FORK_COUNTS, constants.CAT_STARS]:
+                value_type = constants.NUMBER
+            if category == constants.CAT_LICENSE:
+                result = {
+                    constants.PROP_VALUE: value["url"],
+                    constants.PROP_TYPE: constants.LICENSE,
+                    constants.PROP_NAME: value["name"],
+                    constants.PROP_URL: value["url"]
+                }
+                if "spdx_id" in value.keys():
+                    result[constants.PROP_SPDX_ID] = value["spdx_id"]
+            else:
+                result = {
+                    constants.PROP_VALUE: value,
+                    constants.PROP_TYPE: value_type
+                }
+            repository_metadata.add_result(category, result, 1, constants.TECHNIQUE_GITHUB_API)
     # get languages
     if not ignore_api_metadata:
         languages, date = rate_limit_get(filtered_resp['languages_url'], headers=header)
         if "message" in languages:
-            print("Languages Error: " + languages["message"])
+            logging.error("Error while retrieving languages: " + languages["message"])
         else:
             filtered_resp['languages'] = list(languages.keys())
+            for l, s in languages.items():
+                result = {
+                    constants.PROP_VALUE: l,
+                    constants.PROP_NAME: l,
+                    constants.PROP_TYPE: constants.LANGUAGE,
+                    constants.PROP_SIZE: s,
+                }
+                repository_metadata.add_result(constants.CAT_PROGRAMMING_LANGUAGES, result, 1,
+                                               constants.TECHNIQUE_GITHUB_API)
 
-        del filtered_resp['languages_url']
         # get releases
         releases_list, date = rate_limit_get(repo_api_base_url + "/releases",
                                              headers=header)
         if isinstance(releases_list, dict) and 'message' in releases_list.keys():
-            print("Releases Error: " + releases_list['message'])
+            logging.error("Releases Error: " + releases_list['message'])
         else:
-            filtered_resp['releases'] = [do_crosswalk(release, constants.release_crosswalk_table) for release in
-                                         releases_list]
+            release_list_filtered = [do_crosswalk(release, constants.release_crosswalk_table) for release in
+                                     releases_list]
+            for release in release_list_filtered:
+                release_obj = {
+                    constants.PROP_TYPE: constants.RELEASE,
+                    constants.PROP_VALUE: release[constants.PROP_URL]
+                }
+                for category, value in release.items():
+                    if category != constants.AGENT_TYPE:
+                        if category == constants.PROP_AUTHOR:
+                            value = {
+                                constants.PROP_NAME: value,
+                                constants.PROP_TYPE: release[constants.AGENT_TYPE]
+
+                            }
+                        if value != "":
+                            release_obj[category] = value
+                        else:
+                            logging.warning("Ignoring empty value in release for " + category)
+                repository_metadata.add_result(constants.CAT_RELEASES, release_obj, 1,
+                                               constants.TECHNIQUE_GITHUB_API)
     logging.info("Repository information successfully loaded.\n")
-    return filtered_resp, owner, repo_name, default_branch
+    return repository_metadata, owner, repo_name, default_branch
+
+
+def get_path(obj, path):
+    if isinstance(path, list) or isinstance(path, tuple):
+        if len(path) == 1:
+            path = path[0]
+        else:
+            return get_path(obj[path[0]], path[1:])
+
+    if obj is not None and path in obj:
+        return obj[path]
+    else:
+        return None
+
+
+def do_crosswalk(data, crosswalk_table):
+    output = {}
+    for somef_key, path in crosswalk_table.items():
+        value = get_path(data, path)
+        if value is not None:
+            output[somef_key] = value
+        else:
+            logging.error(f"Error: key {path} not present in github repository")
+    return output
 
 
 def download_repository_files(owner, repo_name, default_branch, repo_type, target_dir, repo_ref=None):
@@ -536,12 +519,12 @@ def download_github_files(directory, owner, repo_name, repo_ref):
     """
     # download the repo at the selected branch with the link
     repo_archive_url = f"https://github.com/{owner}/{repo_name}/archive/{repo_ref}.zip"
-    print(f"Downloading {repo_archive_url}")
+    logging.info(f"Downloading {repo_archive_url}")
     repo_download = requests.get(repo_archive_url)
     if repo_download.status_code == 404:
-        print(f"Error: Archive request failed with HTTP {repo_download.status_code}")
+        logging.error(f"Error: Archive request failed with HTTP {repo_download.status_code}")
         repo_archive_url = f"https://github.com/{owner}/{repo_name}/archive/main.zip"
-        print(f"Trying to download {repo_archive_url}")
+        logging.info(f"Trying to download {repo_archive_url}")
         repo_download = requests.get(repo_archive_url)
 
     if repo_download.status_code != 200:
@@ -559,27 +542,15 @@ def download_github_files(directory, owner, repo_name, repo_ref):
         zip_ref.extractall(repo_extract_dir)
 
     repo_folders = os.listdir(repo_extract_dir)
-    # assert (len(repo_folders) == 1)
 
     repo_dir = os.path.join(repo_extract_dir, repo_folders[0])
 
     return repo_dir
-    #return process_repository_files(repo_dir, filtered_resp, constants.RepositoryType.GITHUB,
-    #                                owner, repo_name, repo_ref)
-
-
-# def load_local_repository_metadata(local_repo):
-#     """Function to apply somef to a local repository (already downloaded)"""
-#     filtered_resp = {}
-#     repo_dir = os.path.abspath(local_repo)
-#     text, filtered_resp = process_repository_files(repo_dir, filtered_resp, constants.RepositoryType.LOCAL)
-#     print("Local repository information successfully loaded. \n")
-#     return text, filtered_resp
 
 
 def get_project_id(repository_url):
     """Function to download a repository, given its URL"""
-    print(f"Downloading {repository_url}")
+    logging.info(f"Downloading {repository_url}")
     response = requests.get(repository_url)
     response_str = str(response.content.decode('utf-8'))
     init = response_str.find('\"project_id\":')
@@ -610,5 +581,3 @@ def get_readme_content(readme_url):
     readme = requests.get(readme_url)
     readme_text = readme.content.decode('utf-8')
     return readme_text
-
-
diff --git a/src/somef/process_results.py b/src/somef/process_results.py
new file mode 100644
index 00000000..283721b5
--- /dev/null
+++ b/src/somef/process_results.py
@@ -0,0 +1,63 @@
+import logging
+
+from .utils import constants
+from . import __version__
+import datetime
+
+
+class Result:
+    """
+    Class designed to store results found by SOMEF.
+    For more information, see the output JSON documentation at: docs/output.md
+    """
+    def __init__(self):
+        """
+        init method for the main object to return.
+        By default, the provenance of the execution and the version of somef are also returned
+        """
+        self.results = {
+            constants.PROP_PROVENANCE: {
+                constants.PROP_SOMEF_VERSION:__version__,
+                constants.PROP_SOMEF_SCHEMA_VERSION: "1.0.0",
+                constants.PROP_DATE: datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")
+            },
+        }
+
+    def add_result(self, category, result, confidence, technique, source=""):
+        """
+        Method designed to add a result for a category (i.e., a new description is found)
+        Parameters
+        ----------
+        @param category: one of the fields SOMEF recognizes (see Constants.CAT)
+        @param result: a dictionary with at least value and type
+        @param confidence: confidence in the extraction (0..1)
+        @param technique: technique used in the extraction
+        @param source: source file used for the extraction (if any)
+        """
+        # Sanity check: a result must have a type and a value
+        if isinstance(result,dict)  and \
+                constants.PROP_VALUE in result.keys() and \
+                constants.PROP_TYPE in result.keys():
+            result = {constants.PROP_RESULT: result,
+                      constants.PROP_CONFIDENCE: confidence,
+                      constants.PROP_TECHNIQUE: technique}
+
+            if source != "":
+                result[constants.PROP_SOURCE] = source
+
+            if category in self.results.keys():
+                self.results[category].append(result)
+            else:
+                # Results should always be a list, even if there is only one value (for consistency)
+                self.results[category] = [result]
+        else:
+            logging.error("Tried to add a result without value or type. Discarding it ...")
+
+    def consolidate_results(self):
+        # TO DO: for each category where we may reduce/ improve the results, do so.
+        # For example, here is where we would detect if there are redundant citation files, and we would create a single
+        # publication object.
+
+        # if multiple licenses are found, keep only the outmost one.
+
+        return self.results
\ No newline at end of file
diff --git a/src/somef/regular_expressions.py b/src/somef/regular_expressions.py
index 7928ae2a..368cc9e0 100644
--- a/src/somef/regular_expressions.py
+++ b/src/somef/regular_expressions.py
@@ -1,14 +1,28 @@
+import logging
 import os
 import re
 import markdown
 import requests
 import validators
 from .utils import constants
+from .process_results import Result
 from urllib.parse import urlparse
 
 
-def extract_title(unfiltered_text):
-    """Regexp to extract title (first header) from a repository"""
+def extract_title(unfiltered_text, repository_metadata: Result, readme_source) -> Result:
+    """
+    Regexp to extract title (first header) from a repository
+    Parameters
+    ----------
+    @param unfiltered_text: repo text
+    @param repository_metadata: Result with the extractions so far
+    @param readme_source: url to the file used (for provenance)
+
+    Returns
+    -------
+    @returns a Result including the title (if found)
+
+    """
     html_text = markdown.markdown(unfiltered_text)
     splitted = html_text.split("\n")
     index = 0
@@ -22,7 +36,12 @@ def extract_title(unfiltered_text):
                 output = re.sub(regex, '', line)
             break
         index += 1
-    return output
+    repository_metadata.add_result(constants.CAT_FULL_TITLE,
+                                       {
+                                           constants.PROP_TYPE: constants.STRING,
+                                           constants.PROP_VALUE: output
+                                       }, 1, constants.TECHNIQUE_REGULAR_EXPRESSION, readme_source)
+    return repository_metadata
 
 
 def extract_title_old(unfiltered_text):
@@ -53,33 +72,62 @@ def extract_title_old(unfiltered_text):
     return title
 
 
-def extract_readthedocs(readme_text) -> object:
+def extract_readthedocs(readme_text, repository_metadata: Result, readme_source) -> Result:
     """
     Function to extract readthedocs links from text
     Parameters
     ----------
-    unfiltered_text text readme
+    @param readme_text: raw text of the readme file
+    @param repository_metadata: Result where to deposit the findings of the README
+    @param readme_source: URL of the README file
 
     Returns
     -------
-    Links to the readthedocs documentation
+    @return Result including links to the readthedocs documentation
     """
     readthedocs_links = re.findall(constants.REGEXP_READTHEDOCS, readme_text)
-    print("Extraction of readthedocs links from readme completed.\n")
-    # remove duplicates (links like [readthedocs](readthedocs) are found twice
-    return list(dict.fromkeys(readthedocs_links))
+    name = ""
+    try:
+        name = repository_metadata.results[constants.CAT_NAME][0]
+        name = name[constants.PROP_RESULT][constants.PROP_VALUE]
+    except:
+        pass
+    for link in list(dict.fromkeys(readthedocs_links)):
+        result = {
+            constants.PROP_TYPE: constants.URL,
+            constants.PROP_VALUE: link,
+            constants.PROP_FORMAT: constants.FORMAT_READTHEDOCS
+        }
+        try:
+            # if name of the repo is known then compare against the readthedocs one. Only add it if it's similar/same
+            name_in_link = re.findall('https://([^.]+)\.readthedocs\.io', link)
+            name_in_link = name_in_link[0]
+            if name == "" or name_in_link.lower() == name.lower():
+                repository_metadata.add_result(constants.CAT_DOCUMENTATION, result, 1,
+                                               constants.TECHNIQUE_REGULAR_EXPRESSION, readme_source)
+            else:
+                repository_metadata.add_result(constants.CAT_RELATED_DOCUMENTATION, result, 1,
+                                               constants.TECHNIQUE_REGULAR_EXPRESSION, readme_source)
+        except:
+            # add link as a regular doc link if we cannot retrieve name or there is an error
+            repository_metadata.add_result(constants.CAT_DOCUMENTATION, result, 1,
+                                           constants.TECHNIQUE_REGULAR_EXPRESSION, readme_source)
 
+    return repository_metadata
 
-def extract_support_channels(readme_text):
+
+def extract_support_channels(readme_text, repository_metadata: Result, readme_source) -> Result:
     """
-    Function to extract Gitter Chat, Reddit and Discord links from text
+    Function to extract support channels links from text
     Parameters
     ----------
-    readme_text text readme
+    @param readme_text: raw text of the readme file
+    @param repository_metadata: Result where to deposit the findings of the README
+    @param readme_source: URL of the README file
 
     Returns
     -------
-    Link to the Gitter Chat
+    @return Result including links to the support channels (Gitter, reddit, discord)
     """
     results = []
 
@@ -102,18 +150,41 @@ def extract_support_channels(readme_text):
         repo_status = readme_text[init + 1:end]
         results.append(repo_status)
 
-    return results
+    for link in results:
+        repository_metadata.add_result(constants.CAT_SUPPORT_CHANNELS,
+                                       {
+                                           constants.PROP_TYPE: constants.URL,
+                                           constants.PROP_VALUE: link,
+                                       }, 1, constants.TECHNIQUE_REGULAR_EXPRESSION, readme_source)
+
+    return repository_metadata
+
+
+def extract_repo_status(unfiltered_text, repository_metadata: Result, readme_source) -> Result:
+    """
+    Function takes readme text as input and extracts the repostatus.org badge
 
+    Parameters
+    ----------
+    @param unfiltered_text: Text of the readme
+    @param repository_metadata: Result with all the processed results so far
+    @param readme_source: source to the readme file used
+    """
 
-def extract_repo_status(unfiltered_text):
-    """Extracts the repostatus.org badge from a given text"""
     repo_status = ""
     init = unfiltered_text.find("[![Project Status:")
     if init > 0:
         end = unfiltered_text.find("](", init)
         repo_status = unfiltered_text[init + 3:end]
         repo_status = repo_status.replace("Project Status: ", "")
-    return repo_status
+        short_status = repo_status[0:repo_status.find(" ")].lower()
+        repository_metadata.add_result(constants.CAT_STATUS,
+                                       {
+                                           constants.PROP_TYPE: constants.URL,
+                                           constants.PROP_VALUE: "https://www.repostatus.org/#" + short_status,
+                                           constants.PROP_DESCRIPTION: repo_status
+                                       }, 1, constants.TECHNIQUE_REGULAR_EXPRESSION, readme_source)
+    return repository_metadata
 
 
 def extract_arxiv_links(unfiltered_text):
@@ -133,7 +204,20 @@ def extract_arxiv_links(unfiltered_text):
     return results
 
 
-def extract_wiki_links(unfiltered_text, repo_url):
+def extract_wiki_links(unfiltered_text, repo_url, repository_metadata: Result, readme_source) -> Result:
+    """
+
+    Parameters
+    ----------
+    @param unfiltered_text: text from readme
+    @param repo_url: repository URL
+    @param repository_metadata: results found in the repository so far
+    @param readme_source: readme URL/path
+
+    Returns
+    -------
+    @return a Result with the wiki documentation links found in the README
+    """
     """Extracts wiki links from a given text"""
     links = re.findall(r"\[[^\]]*\]\((.*?)?\)", unfiltered_text)
     output = []
@@ -141,13 +225,15 @@ def extract_wiki_links(unfiltered_text, repo_url):
     for link in links:
         if validators.url(link):
             if link.endswith("/wiki"):
-                output.append(link)
+                if link not in output:
+                    output.append(link)
             else:
                 ends = unfiltered_text.find("(" + link + ")") - 1
                 if unfiltered_text[:ends].find("[") != -1:
                     position = unfiltered_text[:ends].rindex("[") + 1
                     if unfiltered_text[position:ends].find("wiki") >= 0:
-                        output.append(link)
+                        if link not in output:
+                            output.append(link)
             unfiltered_text = unfiltered_text[ends + len(link):]
 
     # to check if a wiki url is available in the repository
@@ -158,208 +244,144 @@ def extract_wiki_links(unfiltered_text, repo_url):
             repo_url = repo_url + "/wiki"
         wiki = requests.get(repo_url, allow_redirects=False)
         if wiki.status_code == 200:
-            output.append(repo_url)
-
-    return list(dict.fromkeys(output))
-
-
-# TO DO: join with image detection in a single method
-# def extract_logo(unfiltered_text, repo_url):
-#     """Extracts logos from a given text"""
-#     logo = ""
-#     index_logo = unfiltered_text.lower().find("![logo]")
-#     if index_logo >= 0:
-#         init = unfiltered_text.find("(", index_logo)
-#         end = unfiltered_text.find(")", init)
-#         logo = unfiltered_text[init + 1:end]
-#     else:
-#         # This is problematic if alt label is used
-#         # TO DO
-#         result = [_.start() for _ in re.finditer("<img src=", unfiltered_text)]
-#         if len(result) > 0:
-#             for index_img in result:
-#                 init = unfiltered_text.find("\"", index_img)
-#                 end = unfiltered_text.find("\"", init + 1)
-#                 img = unfiltered_text[init + 1:end]
-#                 if img.find("logo") > 0:
-#                     logo = img
-#     # processing for those github links that do not resolve correctly
-#     if logo.startswith("http"):
-#         if logo.find("github.com") > 0:
-#             logo = logo.replace("/blob", "")
-#             logo = logo.replace("github.com", "raw.githubusercontent.com")
-#     #processing for those logos that are relative paths
-#     if logo != "" and not logo.startswith("http") and repo_url is not None:
-#         if repo_url.find("github.com") > 0:
-#             if repo_url.find("/tree/") > 0:
-#                 repo_url = repo_url.replace("/tree/", "/")
-#             else:
-#                 repo_url = repo_url + "/master/"
-#             repo_url = repo_url.replace("github.com", "raw.githubusercontent.com")
-#             if not repo_url.endswith("/"):
-#                 repo_url = repo_url + "/"
-#             logo = repo_url + logo
-#         else:
-#             if not repo_url.endswith("/"):
-#                 repo_url = repo_url + "/"
-#             logo = repo_url + logo
-#     print(logo)
-#     return logo
-
-
-def extract_images(unfiltered_text, repo_url, local_repo):
-    """Extracts logos and images from a given text"""
+            # sometimes the repo starts with caps
+            links_in_list = [x.lower() for x in output]
+            if repo_url.lower() not in links_in_list:
+                output.append(repo_url)
+
+    for link in output:
+        repository_metadata.add_result(constants.CAT_DOCUMENTATION,
+                                       {
+                                           constants.PROP_TYPE: constants.URL,
+                                           constants.PROP_VALUE: link,
+                                           constants.PROP_FORMAT: constants.FORMAT_WIKI
+                                       },
+                                       1, constants.TECHNIQUE_REGULAR_EXPRESSION, readme_source)
+    return repository_metadata
+
+
+def extract_images(unfiltered_text, repo_url, local_repo, repository_metadata: Result, readme_source, def_branch) -> Result:
+    """
+    Function that takes readme text as input and extracts logos and images
+
+    Parameters
+    ----------
+    @param unfiltered_text: Text of the readme
+    @param repo_url: Repository URL
+    @param local_repo: Local repo path
+    @param repository_metadata: Result with all the processed results so far
+    @param readme_source: source to the readme file used
+    @param def_branch: default branch of the repo
+
+    Returns
+    -------
+    A Result object with the logos and images from the given text
+    """
     logo = ""
-    has_logo = False
     images = []
-    repo = False
     repo_name = ""
     if repo_url is not None and repo_url != "":
         url = urlparse(repo_url)
         path_components = url.path.split('/')
         repo_name = path_components[2]
-        repo = True
 
     html_text = markdown.markdown(unfiltered_text)
     img_md = re.findall(constants.REGEXP_IMAGES, html_text)
-    result = [_.start() for _ in re.finditer("<img ", html_text)]
+    img_html = [_.start() for _ in re.finditer("<img ", html_text)]
     for img in img_md:
-        img = img[1] # the 0 position is the name used in the link
+        img = img[1]  # the 0 position is the name used in the link
         # if the image contains jitpack.io, the element is not processed
         if img.find("jitpack.io") > 0 or img.find("/badge") >= 0 or img.find("/travis-ci.") >= 0 \
                 or img.find("img.shields.io") >= 0:
-            None
-        elif not has_logo and repo:
+            pass
+        elif logo == "" and repo_url is not None:
             start = img.rindex("/")
             if img.find(repo_name, start) > 0:
-                logo = rename_github_image(img, repo_url, local_repo)
-                has_logo = True
+                logo = rename_github_image(img, repo_url, local_repo, def_branch)
             elif get_alt_text_md(html_text, img) == repo_name or get_alt_text_md(html_text, img).upper() == "LOGO":
-                logo = rename_github_image(img, repo_url, local_repo)
-                has_logo = True
+                logo = rename_github_image(img, repo_url, local_repo, def_branch)
             else:
                 start = img.rindex("/")
                 if img.upper().find("LOGO", start) > 0:
-                    logo = rename_github_image(img, repo_url, local_repo)
-                    has_logo = True
+                    logo = rename_github_image(img, repo_url, local_repo, def_branch)
                 else:
-                    images.append(rename_github_image(img, repo_url, local_repo))
+                    images.append(rename_github_image(img, repo_url, local_repo, def_branch))
         else:
-            images.append(rename_github_image(img, repo_url, local_repo))
-    for index_img in result:
+            images.append(rename_github_image(img, repo_url, local_repo, def_branch))
+
+    for index_img in img_html:
         init = html_text.find("src=\"", index_img)
         end = html_text.find("\"", init + 5)
         img = html_text[init + 5:end]
         # if the image contains jitpack.io, the element is not processed
         if img.find("jitpack.io") > 0 or img.find("/badge") >= 0 or img.find("/travis-ci.") >= 0 \
                 or img.find("img.shields.io") >= 0:
-            None
-        elif not has_logo and repo:
+            pass
+        elif logo == "" and repo_url is not None:
             start = 0
             if img.find("/") > 0:
                 start = img.rindex("/")
             image_name = img[start:]
             if image_name.find(repo_name) > 0 or image_name.upper().find("LOGO") > 0:
-                logo = rename_github_image(img, repo_url, local_repo)
-                has_logo = True
+                logo = rename_github_image(img, repo_url, local_repo, def_branch)
             elif get_alt_text_img(html_text, index_img) == repo_name or get_alt_text_img(html_text,
                                                                                          index_img).upper() == "LOGO":
-                logo = rename_github_image(img, repo_url, local_repo)
-                has_logo = True
+                logo = rename_github_image(img, repo_url, local_repo, def_branch)
             else:
-                images.append(rename_github_image(img, repo_url, local_repo))
+                images.append(rename_github_image(img, repo_url, local_repo, def_branch))
         else:
             start = img.rindex("/")
             if img.upper().find("LOGO", start) > 0:
-                logo = rename_github_image(img, repo_url, local_repo)
-                has_logo = True
+                logo = rename_github_image(img, repo_url, local_repo, def_branch)
             else:
-                images.append(rename_github_image(img, repo_url, local_repo))
-
-    return logo, images
-
-
-# TO DO: join with logo detection
-# def extract_images_old(unfiltered_text, repo_url):
-#     """Extracts images from a given text"""
-#     logo = ""
-#     images = []
-#     html_text = markdown.markdown(unfiltered_text)
-#     # print(html_text)
-#     result = [_.start() for _ in re.finditer("<img ", html_text)]
-#     if len(result) > 0:
-#         for index_img in result:
-#             init = html_text.find("src=\"", index_img)
-#             end = html_text.find("\"", init + 5)
-#             img = html_text[init + 5:end]
-#             if img.find("logo") < 0:
-#                 if not img.startswith("http") and repo_url is not None:
-#                     if repo_url.find("/tree/") > 0:
-#                         repo_url = repo_url.replace("/tree/", "/")
-#                     else:
-#                         repo_url = repo_url + "/master/"
-#                     repo_url = repo_url.replace("github.com", "raw.githubusercontent.com")
-#                     if not repo_url.endswith("/"):
-#                         repo_url = repo_url + "/"
-#                     img = repo_url + img
-#                 images.append(img)
-#             else:
-#                 if not img.startswith("http") and repo_url is not None:
-#                     if repo_url.find("/tree/") > 0:
-#                         repo_url = repo_url.replace("/tree/", "/")
-#                     else:
-#                         repo_url = repo_url + "/master/"
-#                     repo_url = repo_url.replace("github.com", "raw.githubusercontent.com")
-#                     if not repo_url.endswith("/"):
-#                         repo_url = repo_url + "/"
-#                     img = repo_url + img
-#                 logo = img
-#
-#     if logo == "" and repo_url != "" and repo_url != None:
-#         print(repo_url)
-#         url = urlparse(repo_url)
-#         path_components = url.path.split('/')
-#         repo_name = path_components[2]
-#
-#         for image in images:
-#             start = image.rindex("/")
-#             if image.find(repo_name, start) > 0:
-#                 logo = image
-#                 images.remove(image)
-#                 break
-#
-#     return logo, images
-
-
-# def extract_support(unfiltered_text):
-#     """Extracts support channels (reddit, discord, gitter) from a given text"""
-#     results = []
-#     init = unfiltered_text.find(constants.REGEXP_REDDIT)
-#     if init > 0:
-#         end = unfiltered_text.find(")", init)
-#         repo_status = unfiltered_text[init + 1:end]
-#         results.append(repo_status)
-#
-#     init = unfiltered_text.find(constants.REGEXP_DISCORD)
-#     if init > 0:
-#         end = unfiltered_text.find(")", init)
-#         repo_status = unfiltered_text[init + 1:end]
-#         results.append(repo_status)
-#
-#     return results
-
-
-def extract_package_distributions(unfiltered_text):
-    """Extracts package distributions from a given text"""
+                images.append(rename_github_image(img, repo_url, local_repo, def_branch))
+    if logo != "":
+        repository_metadata.add_result(constants.CAT_LOGO,
+                                       {
+                                           constants.PROP_TYPE: constants.URL,
+                                           constants.PROP_VALUE: logo
+                                       }, 1, constants.TECHNIQUE_REGULAR_EXPRESSION, readme_source)
+    for image in images:
+        repository_metadata.add_result(constants.CAT_IMAGE,
+                                       {
+                                           constants.PROP_TYPE: constants.URL,
+                                           constants.PROP_VALUE: image
+                                       }, 1, constants.TECHNIQUE_REGULAR_EXPRESSION, readme_source)
+    return repository_metadata
+
+
+def extract_package_distributions(unfiltered_text, repository_metadata: Result, readme_source) -> Result:
+    """
+    Function that takes readme text as input (cleaned from markdown notation) and runs a regex expression on top of it.
+    Extracts package distributions from a given text
+
+    Parameters
+    ----------
+    @param unfiltered_text: Text of the readme
+    @param repository_metadata: Result with all the processed results so far
+    @param readme_source: source to the readme file used
+
+    Returns
+    -------
+    A Result object with the package distributions found in the repo (Python packages for now)
+    """
     output = ""
     index_package_distribution = unfiltered_text.find(constants.REGEXP_PYPI)
+    # If not found, we try again with the other regexp
+    if index_package_distribution <= 0:
+        index_package_distribution = unfiltered_text.find(constants.REGEXP_PYPI_2)
     if index_package_distribution > 0:
         init = unfiltered_text.find(")](", index_package_distribution)
         end = unfiltered_text.find(")", init + 3)
         package_distribution = unfiltered_text[init + 3:end]
         output = requests.get(package_distribution).url
+        repository_metadata.add_result(constants.CAT_PACKAGE_DISTRIBUTION,
+                                       {
+                                           constants.PROP_TYPE: constants.URL,
+                                           constants.PROP_VALUE: output
+                                       }, 1, constants.TECHNIQUE_REGULAR_EXPRESSION, readme_source)
 
-    return output
+    return repository_metadata
 
 
 def extract_colab_links(text):
@@ -405,7 +427,7 @@ def remove_links_images(text):
     links = re.findall(constants.REGEXP_LINKS, text)
     for link in links:
         link_text = link[1]
-        pos = text.find("("+link_text+")")
+        pos = text.find("(" + link_text + ")")
         if pos != -1:
             init = text[:pos].rindex("[")
             end = text[pos:].index(")")
@@ -416,64 +438,116 @@ def remove_links_images(text):
     return text.strip()
 
 
-def extract_bibtex(readme_text) -> object:
+def extract_bibtex(readme_text, repository_metadata: Result, readme_source) -> Result:
     """
     Function takes readme text as input (cleaned from markdown notation) and runs a regex expression on top of it.
     Returns list of bibtex citations
-    """
-    citations = re.findall(constants.REGEXP_BIBTEX, readme_text)
-    return citations
 
+    Parameters
+    ----------
+    @param readme_text: Text of the readme
+    @param repository_metadata: Result with all the processed results so far
+    @param readme_source: source to the readme file used
 
-def extract_dois(readme_text) -> object:
+    Returns
+    -------
+    @returns Result object with the bibtex associated with this software component
+    """
+    citations = re.findall(constants.REGEXP_BIBTEX, readme_text)
+    for c in citations:
+        # try to detect the doi with a regular expression. We should improve this to load an existing library
+        result = {
+            constants.PROP_VALUE: c,
+            constants.PROP_TYPE: constants.TEXT_EXCERPT,
+            constants.PROP_FORMAT: constants.FORMAT_BIB
+        }
+        doi_text = ""
+        if c.find('https://doi.org/') >= 0 or c.find('doi ') >= 0:
+            text_citation = c
+            if text_citation.find("https://doi.org/") >= 0:
+                doi_pos = text_citation.find("doi.org/")
+                starts = text_citation[:doi_pos].rindex("http")
+                ends = text_citation[starts:].find("}")
+                doi_text = text_citation[starts:starts + ends]
+            elif text_citation.find("doi") >= 0:
+                doi_pos = text_citation.find("doi")
+                starts = text_citation[doi_pos:].find("{")
+                ends = text_citation[starts + doi_pos:].find("}")
+                doi_text = "https://doi.org/" + text_citation[starts + doi_pos + 1:doi_pos + starts + ends]
+        if doi_text != "":
+            result[constants.PROP_DOI] = doi_text
+        repository_metadata.add_result(constants.CAT_CITATION, result, 1, constants.TECHNIQUE_REGULAR_EXPRESSION,
+                                       readme_source)
+    return repository_metadata
+
+
+def extract_doi_badges(readme_text, repository_metadata: Result, source) -> Result:
     """
     Function that takes the text of a readme file and searches if there are any DOIs badges.
     Parameters
     ----------
-    readme_text Text of the readme
-
+    @param readme_text: Text of the readme
+    @param repository_metadata: Result with all the findings in the repo
+    @param source: source file on top of which the extraction is performed (provenance)
     Returns
     -------
-    DOIs/identifiers associated with this software component
+    @returns Result with the DOI badges found
     """
     # regex = r'\[\!\[DOI\]([^\]]+)\]\(([^)]+)\)'
     # regex = r'\[\!\[DOI\]\(.+\)\]\(([^)]+)\)'
-    dois = re.findall(constants.REGEXP_DOI, readme_text)
-    print("Extraction of DOIS from readme completed.\n")
-    return dois
-
-
-def extract_binder_links(readme_text) -> object:
+    doi_badges = re.findall(constants.REGEXP_DOI, readme_text)
+    # The identifier is in position 1. Position 0 is the badge id, which we don't want to export
+    for doi in doi_badges:
+        repository_metadata.add_result(constants.CAT_IDENTIFIER,
+                                       {
+                                           constants.PROP_TYPE: constants.URL,
+                                           constants.PROP_VALUE: doi[1]
+                                       }, 1, constants.TECHNIQUE_REGULAR_EXPRESSION, source)
+    return repository_metadata
+
+
+def extract_binder_links(readme_text, repository_metadata: Result, source) -> Result:
     """
-    Function that does a regex to extract binder links used as reference in the readme.
-    There could be multiple binder links for one reprository
+    Function that does a regex to extract binder and colab links used as reference in the readme.
+    There could be multiple binder links for one repository
     Parameters
     ----------
-    readme_text
-
+    @param readme_text: Text of the readme
+    @param repository_metadata: Result with all the findings in the repo
+    @param source: source file on top of which the extraction is performed (provenance)
     Returns
     -------
     Links with binder notebooks/scripts that are ready to be executed.
     """
-    binder_links = re.findall(constants.REGEXP_BINDER, readme_text)
-    print("Extraction of Binder links from readme completed.\n")
+    links = re.findall(constants.REGEXP_BINDER, readme_text, re.IGNORECASE)
+    binder_links = [result[1] for result in links]
     # extract binder links and remove duplicates
     binder_links += extract_colab_links(readme_text)
-    return list(dict.fromkeys(binder_links))
+    for link in list(dict.fromkeys(binder_links)):
+        repository_metadata.add_result(constants.CAT_EXECUTABLE_EXAMPLE,
+                                       {
+                                           constants.PROP_TYPE: constants.URL,
+                                           constants.PROP_VALUE: link
+                                       }, 1, constants.TECHNIQUE_REGULAR_EXPRESSION, source)
+    return repository_metadata
 
 
-def rename_github_image(img, repo_url, local_repo):
+def rename_github_image(img, repo_url, local_repo, default_branch):
     """Renames GitHub image links so they can be accessed raw"""
     if img.startswith("http"):
+        if "/raw/" in img:
+            # raw link is already in there: do nothing
+            return img
         if img.find("github.com") > 0:
             img = img.replace("/blob", "")
             img = img.replace("github.com", "raw.githubusercontent.com")
-    if not img.startswith("http") and ((repo_url is not None and repo_url != "") or (local_repo is not None and local_repo != "")):
+    if not img.startswith("http") and (
+            (repo_url is not None and repo_url != "") or (local_repo is not None and local_repo != "")):
         if repo_url is not None and repo_url != "":
             if repo_url.find("/tree/") > 0:
                 repo_url = repo_url.replace("/tree/", "/")
             else:
-                repo_url = repo_url + "/master/"
+                repo_url = repo_url + "/"+default_branch+"/"
             repo_url = repo_url.replace("github.com", "raw.githubusercontent.com")
             if not repo_url.endswith("/"):
                 repo_url = repo_url + "/"
diff --git a/src/somef/somef_cli.py b/src/somef/somef_cli.py
index a571fa14..cccdfbd5 100644
--- a/src/somef/somef_cli.py
+++ b/src/somef/somef_cli.py
@@ -7,267 +7,15 @@
 from os import path
 from . import header_analysis, regular_expressions, process_repository, configuration, process_files, \
     supervised_classification
+from .process_results import Result
 from .utils import constants, markdown_utils
 from .parser import mardown_parser, create_excerpts
 from .export.data_to_graph import DataGraph
 from .export import json_export
 
 
-def is_in_excerpts_headers(text, set_excerpts):
-    """
-    Function that checks if some text is included in a set of excerpts
-    Parameters
-    ----------
-    text: text to look for
-    set_excerpts: existing set of excerpts
-
-    Returns
-    -------
-    True if the text is included in the excerpts, False otherwise.
-    """
-    set_text = set(text.split())
-    for excerpt in set_excerpts:
-        set_excerpt = set(excerpt.split())
-        if set_text.issubset(set_excerpt):
-            return True, excerpt
-
-    return False, None
-
-
-def extract_categories_using_header(repo_data):
-    """
-    Function that adds category information extracted using header information
-    Parameters
-    ----------
-    repo_data data to use the header analysis
-
-    Returns
-    -------
-    Returns json with the information added.
-    """
-    print("Extracting information using headers")
-    # this is a hack because if repo_data is "" this errors out
-    if repo_data is None or len(repo_data) == 0:
-        return {}, []
-    try:
-        header_info, string_list = header_analysis.extract_categories_using_headers(repo_data)
-        print("Information extracted. \n")
-        return header_info, string_list
-    except:
-        print("Error while extracting headers: ", sys.exc_info()[0])
-        return {}, [repo_data]
-
-
-def merge(header_predictions, predictions, citations, citation_file_text, dois, binder_links, long_title,
-          readthedocs_links, repo_status, arxiv_links, logo, images, support_channels, package_distribution,
-          wiki_links, category):
-    """
-    Function that takes the predictions using header information, classifier and bibtex/doi parser
-    Parameters
-    ----------
-    header_predictions: extraction of common headers and their contents
-    wiki_links: links to wikis
-    package_distribution: packages that appear in the readme
-    support_channels: like gitter, discord, etc.
-    images: included in the readme
-    logo: included in the readme
-    arxiv_links: links to arxiv papers
-    repo_status: repostatus.org badges
-    readthedocs_links: documentation links
-    long_title: title of the repository
-    binder_links: links to binder notebooks
-    citation_file_text: text of the citation file
-    header_predictions: predicted headers
-    predictions: predictions from classifiers (description, installation instructions, invocation, citation)
-    citations: bibtex citations
-    dois: identifiers found in readme Zenodo DOIs, or other
-    category: prediction of the category of the given repo
-    Returns
-    -------
-    Combined predictions and results of the extraction process
-    """
-    print("Merge prediction using header information, classifier and bibtex and doi parsers")
-    if long_title:
-        predictions['longTitle'] = {'excerpt': long_title, 'confidence': [1.0],
-                                    'technique': 'Regular expression'}
-    for i in range(len(citations)):
-        if 'citation' not in predictions.keys():
-            predictions['citation'] = []
-        if citations[i].find('https://doi.org/') >= 0 or citations[i].find('doi ') >= 0:
-            doi_text = ""
-            text_citation = citations[i]
-            if text_citation.find("https://doi.org/") >= 0:
-                doi_pos = text_citation.find("doi.org/")
-                starts = text_citation[:doi_pos].rindex("http")
-                ends = text_citation[starts:].find("}")
-                doi_text = text_citation[starts:starts + ends]
-            elif text_citation.find("doi") >= 0:
-                doi_pos = text_citation.find("doi")
-                starts = text_citation[doi_pos:].find("{")
-                ends = text_citation[starts + doi_pos:].find("}")
-                doi_text = "https://doi.org/" + text_citation[starts + doi_pos + 1:doi_pos + starts + ends]
-            predictions['citation'].append({'excerpt': citations[i], 'confidence': [1.0],
-                                            'technique': 'Regular expression', 'doi': doi_text,
-                                            'format': 'bibtex'})
-        else:
-            predictions['citation'].append({'excerpt': citations[i], 'confidence': [1.0],
-                                            'technique': 'Regular expression', 'format': 'bibtex'})
-    if len(citation_file_text) != 0:
-        if 'citation' not in predictions.keys():
-            predictions['citation'] = []
-        predictions['citation'].append({'excerpt': citation_file_text, 'confidence': [1.0],
-                                        'technique': 'File Exploration', 'format': 'citation file format'})
-    if len(dois) != 0:
-        predictions['identifier'] = []
-        for identifier in dois:
-            # The identifier is in position 1. Position 0 is the badge id, which we don't want to export
-            predictions['identifier'].append({'excerpt': identifier[1], 'confidence': [1.0],
-                                              'technique': 'Regular expression'})
-    if len(binder_links) != 0:
-        predictions['executableExample'] = {'excerpt': binder_links, 'confidence': [1.0],
-                                            'technique': 'Regular expression'}
-    if len(repo_status) != 0:
-        predictions['repoStatus'] = {
-            'excerpt': "https://www.repostatus.org/#" + repo_status[0:repo_status.find(" ")].lower(),
-            'description': repo_status,
-            'confidence': [1.0],
-            'technique': 'Regular expression'}
-
-    if len(arxiv_links) != 0:
-        predictions['arxivLinks'] = {'excerpt': arxiv_links, 'confidence': [1.0],
-                                     'technique': 'Regular expression'}
-
-    if len(logo) != 0:
-        predictions['logo'] = {'excerpt': logo, 'confidence': [1.0],
-                               'technique': 'Regular expression'}
-
-    if len(images) != 0:
-        badges = []
-        for image in images:
-            if image.find('badge') >= 0:
-                badges.append(image)
-        for badge in badges:
-            images.remove(badge)
-        if len(images) > 0:
-            predictions['image'] = []
-            for image in images:
-                predictions['image'].append({'excerpt': image, 'confidence': [1.0],
-                                             'technique': 'Regular expression'})
-
-    if len(support_channels) != 0:
-        predictions['supportChannels'] = {'excerpt': support_channels, 'confidence': [1.0],
-                                          'technique': 'Regular expression'}
-
-    if len(package_distribution) != 0:
-        predictions['packageDistribution'] = {'excerpt': package_distribution, 'confidence': [1.0],
-                                              'technique': 'Regular expression'}
-
-    for i in range(len(readthedocs_links)):
-        if 'documentation' not in predictions.keys():
-            predictions['documentation'] = []
-        predictions['documentation'].append({'excerpt': readthedocs_links[i], 'confidence': [1.0],
-                                             'technique': 'Regular expression', 'type': 'readthedocs'})
-
-    for i in range(len(wiki_links)):
-        if 'documentation' not in predictions.keys():
-            predictions['documentation'] = []
-        predictions['documentation'].append({'excerpt': wiki_links[i], 'confidence': [1.0],
-                                             'technique': 'Regular expression', 'type': 'wiki'})
-
-    if category:
-        predictions['category'] = category
-
-    for headers in header_predictions:
-        if headers not in predictions.keys():
-            predictions[headers] = header_predictions[headers]
-        else:
-            for h in header_predictions[headers]:
-                predictions[headers].insert(0, h)
-    print("Merging successful. \n")
-    return predictions
-
-
-def format_output(git_data, repo_data, repo_type):
-    """
-    Function takes metadata, readme text predictions, bibtex citations and path to the output file
-    Parameters
-    ----------
-    git_data GitHub obtained data
-    repo_data Data extracted from the code repo by SOMEF
-
-    Returns
-    -------
-    json representation of the categories found in file
-    """
-    text_technique = 'GitHub API'
-    if repo_type is constants.RepositoryType.GITLAB:
-        text_technique = 'GitLab API'
-    print("formatting output")
-
-    for i in git_data.keys():
-        if i == 'description':
-            if 'description' not in repo_data.keys():
-                repo_data['description'] = []
-            if git_data[i] != "":
-                repo_data['description'].append(
-                    {'excerpt': git_data[i], 'confidence': [1.0], 'technique': text_technique})
-        else:
-            keys = repo_data.keys
-            if i in constants.file_exploration:
-                if i == 'hasExecutableNotebook':
-                    repo_data[i] = {'excerpt': git_data[i], 'confidence': [1.0], 'technique': 'File Exploration',
-                                    'format': 'jupyter notebook'}
-                elif i == 'hasBuildFile':
-                    docker_files = []
-                    docker_compose = []
-                    for data in git_data[i]:
-                        if data.lower().endswith('docker-compose.yml'):
-                            docker_compose.append(data)
-                        else:
-                            docker_files.append(data)
-                    repo_data[i] = []
-                    if len(docker_files) > 0:
-                        repo_data[i].append({'excerpt': docker_files, 'confidence': [1.0],
-                                             'technique': 'File Exploration',
-                                             'format': 'Docker file'})
-                    if len(docker_compose) > 0:
-                        repo_data[i].append({'excerpt': docker_compose, 'confidence': [1.0],
-                                             'technique': 'File Exploration',
-                                             'format': 'Docker compose file'})
-                else:
-                    if i in repo_data:
-                        repo_data[i].append(
-                            {'excerpt': git_data[i], 'confidence': [1.0], 'technique': 'File Exploration'})
-                    else:
-                        repo_data[i] = {'excerpt': git_data[i], 'confidence': [1.0], 'technique': 'File Exploration'}
-            elif git_data[i] != "" and git_data[i] != []:
-                repo_data[i] = {'excerpt': git_data[i], 'confidence': [1.0], 'technique': text_technique}
-    # remove empty categories from json
-    return remove_empty_elements(repo_data)
-
-
-def remove_empty_elements(d):
-    """recursively remove empty lists, empty dicts, or None elements from a dictionary"""
-
-    def empty(x):
-        return x is None or x == {} or x == []
-
-    if not isinstance(d, (dict, list)):
-        return d
-    elif isinstance(d, list):
-        return [v for v in (remove_empty_elements(v) for v in d) if not empty(v)]
-    else:
-        return {k: v for k, v in ((k, remove_empty_elements(v)) for k, v in d.items()) if not empty(v)}
-
-
-def save_json(git_data, repo_data, outfile):
-    """Performs some combinations and saves the final json Object in output file"""
-    repo_data = format_output(git_data, repo_data)
-    json_export.save_json_output(repo_data, outfile, None)
-
-
 def cli_get_data(threshold, ignore_classifiers, repo_url=None, doc_src=None, local_repo=None,
-                 ignore_github_metadata=False, readme_only=False, keep_tmp=None):
+                 ignore_github_metadata=False, readme_only=False, keep_tmp=None) -> Result:
     """
     Main function to get the data through the command line
     Parameters
@@ -283,20 +31,21 @@ def cli_get_data(threshold, ignore_classifiers, repo_url=None, doc_src=None, loc
 
     Returns
     -------
-    @return: JSON file with the results found by SOMEF.
+    @return: Dictionary with the results found by SOMEF, formatted as a Result object.
     """
     file_paths = configuration.get_configuration_file()
     repo_type = constants.RepositoryType.GITHUB
-    # TO DO Create a new result (when everything works)
-    # result = ProcessedResult()
+    repository_metadata = Result()
+    def_branch = "main"
     if repo_url is not None:
         try:
             if repo_url.rfind("gitlab.com") > 0:
                 repo_type = constants.RepositoryType.GITLAB
             repository_metadata, owner, repo_name, def_branch = process_repository.load_online_repository_metadata(
-                                                                                            repo_url,
-                                                                                            ignore_github_metadata,
-                                                                                            repo_type)
+                repository_metadata,
+                repo_url,
+                ignore_github_metadata,
+                repo_type)
             # download files and obtain path to download folder
             if readme_only:
                 # download readme only with the information above
@@ -306,90 +55,85 @@ def cli_get_data(threshold, ignore_classifiers, repo_url=None, doc_src=None, loc
                 os.makedirs(keep_tmp, exist_ok=True)
                 local_folder = process_repository.download_repository_files(owner, repo_name, def_branch, repo_type,
                                                                             keep_tmp, repo_url)
-                readme_text, repository_metadata = process_files.process_repository_files(local_folder,
-                                                                                          repository_metadata,
-                                                                                          repo_type, owner, repo_name,
-                                                                                          def_branch)
+                readme_text, full_repository_metadata = process_files.process_repository_files(local_folder,
+                                                                                               repository_metadata,
+                                                                                               repo_type, owner,
+                                                                                               repo_name,
+                                                                                               def_branch)
             else:  # Use a temp directory
                 with tempfile.TemporaryDirectory() as temp_dir:
                     local_folder = process_repository.download_repository_files(owner, repo_name, def_branch, repo_type,
                                                                                 temp_dir, repo_url)
-                    readme_text, repository_metadata = process_files.process_repository_files(local_folder,
-                                                                                              repository_metadata,
-                                                                                              repo_type, owner,
-                                                                                              repo_name,
-                                                                                              def_branch)
+                    readme_text, full_repository_metadata = process_files.process_repository_files(local_folder,
+                                                                                                   repository_metadata,
+                                                                                                   repo_type, owner,
+                                                                                                   repo_name,
+                                                                                                   def_branch)
             if readme_text == "":
                 logging.warning("README document does not exist in the target repository")
         except process_repository.GithubUrlError:
             logging.error("Error processing the target repository")
-            return None
+            return repository_metadata
     elif local_repo is not None:
         try:
-            readme_text, repository_metadata = process_files.process_repository_files(local_repo, {}, repo_type)
+            readme_text, full_repository_metadata = process_files.process_repository_files(local_repo,
+                                                                                           repository_metadata,
+                                                                                           repo_type)
             if readme_text == "":
                 logging.warning("Warning: README document does not exist in the local repository")
         except process_repository.GithubUrlError:
             logging.error("Error processing the input repository")
-            return None
+            return repository_metadata
     else:
         if doc_src is None or not path.exists(doc_src):
             logging.error("Error processing the input repository")
             sys.exit()
         with open(doc_src, 'r', encoding="UTF-8") as doc_fh:
             readme_text = doc_fh.read()
-        repository_metadata = {}
     try:
         unfiltered_text = readme_text
-        header_predictions, string_list = extract_categories_using_header(unfiltered_text)
+        repository_metadata, string_list = header_analysis.extract_categories(unfiltered_text, repository_metadata)
         readme_text = markdown_utils.unmark(readme_text)
-        category = supervised_classification.run_category_classification(unfiltered_text, threshold)
-        excerpts = create_excerpts.create_excerpts(string_list)
-        if ignore_classifiers or unfiltered_text == '':
-            predictions = {}
-        else:
+        if not ignore_classifiers and unfiltered_text != '':
+            repository_metadata = supervised_classification.run_category_classification(unfiltered_text, threshold,
+                                                                                        repository_metadata)
+            excerpts = create_excerpts.create_excerpts(string_list)
             excerpts_headers = mardown_parser.extract_text_excerpts_header(unfiltered_text)
             header_parents = mardown_parser.extract_headers_parents(unfiltered_text)
             score_dict = supervised_classification.run_classifiers(excerpts, file_paths)
-            predictions = supervised_classification.classify(score_dict, threshold, excerpts_headers, header_parents)
-        if readme_text != '':
-            citations = regular_expressions.extract_bibtex(readme_text)
-            citation_file_text = ''
-            if 'citation' in repository_metadata.keys():
-                citation_file_text = repository_metadata['citation']
-                del repository_metadata['citation']
-            dois = regular_expressions.extract_dois(unfiltered_text)
-            binder_links = regular_expressions.extract_binder_links(unfiltered_text)
-            title = regular_expressions.extract_title(unfiltered_text)
-            readthedocs_links = regular_expressions.extract_readthedocs(unfiltered_text)
-            repo_status = regular_expressions.extract_repo_status(unfiltered_text)
-            arxiv_links = regular_expressions.extract_arxiv_links(unfiltered_text)
-            wiki_links = regular_expressions.extract_wiki_links(unfiltered_text, repo_url)
-            logo, images = regular_expressions.extract_images(unfiltered_text, repo_url, local_repo)
-            support_channels = regular_expressions.extract_support_channels(unfiltered_text)
-            package_distribution = regular_expressions.extract_package_distributions(unfiltered_text)
-        else:
-            citations = []
-            citation_file_text = ""
-            dois = []
-            binder_links = []
-            title = ""
-            readthedocs_links = []
-            repo_status = ""
-            arxiv_links = []
-            wiki_links = []
-            logo = ""
-            images = []
-            support_channels = []
-            package_distribution = ""
-
-        predictions = merge(header_predictions, predictions, citations, citation_file_text, dois, binder_links, title,
-                            readthedocs_links, repo_status, arxiv_links, logo, images, support_channels,
-                            package_distribution, wiki_links, category)
-        return format_output(repository_metadata, predictions, repo_type)
-    except Exception:
-        logging.error("Error processing repository")
-        return None
+            repository_metadata = supervised_classification.classify(score_dict, threshold, excerpts_headers,
+                                                                     header_parents, repository_metadata)
+        if readme_text != "":
+            try:
+                readme_source = repository_metadata.results[constants.CAT_README_URL][0]
+                readme_source = readme_source[constants.PROP_RESULT][constants.PROP_VALUE]
+            except:
+                readme_source = "README.md"
+            repository_metadata = regular_expressions.extract_bibtex(unfiltered_text, repository_metadata, readme_source)
+            repository_metadata = regular_expressions.extract_doi_badges(unfiltered_text, repository_metadata,
+                                                                         readme_source)
+            repository_metadata = regular_expressions.extract_title(unfiltered_text, repository_metadata, readme_source)
+            repository_metadata = regular_expressions.extract_binder_links(unfiltered_text, repository_metadata,
+                                                                           readme_source)
+            repository_metadata = regular_expressions.extract_readthedocs(unfiltered_text, repository_metadata,
+                                                                          readme_source)
+            repository_metadata = regular_expressions.extract_repo_status(unfiltered_text, repository_metadata,
+                                                                          readme_source)
+            repository_metadata = regular_expressions.extract_wiki_links(unfiltered_text, repo_url, repository_metadata,
+                                                                         readme_source)
+            repository_metadata = regular_expressions.extract_support_channels(unfiltered_text, repository_metadata,
+                                                                               readme_source)
+            repository_metadata = regular_expressions.extract_package_distributions(unfiltered_text,
+                                                                                    repository_metadata,
+                                                                                    readme_source)
+            repository_metadata = regular_expressions.extract_images(unfiltered_text, repo_url, local_repo,
+                                                                     repository_metadata, readme_source, def_branch)
+            logging.info("Completed extracting regular expressions")
+            return repository_metadata
+
+    except Exception as e:
+        logging.error("Error processing repository " + str(e))
+        return repository_metadata
 
 
 def run_cli_document(doc_src, threshold, output):
@@ -457,7 +201,7 @@ def run_cli(*,
                                      doc_src=doc_src, keep_tmp=keep_tmp)
 
     if output is not None:
-        json_export.save_json_output(repo_data, output, missing, pretty=pretty)
+        json_export.save_json_output(repo_data.results, output, missing, pretty=pretty)
 
     if graph_out is not None:
         logging.info("Generating Knowledge Graph")
@@ -473,4 +217,4 @@ def run_cli(*,
             out_file.write(data_graph.g.serialize(format=graph_format, encoding="UTF-8"))
 
     if codemeta_out is not None:
-        json_export.save_codemeta_output(repo_data, codemeta_out, pretty=pretty)
+        json_export.save_codemeta_output(repo_data.results, codemeta_out, pretty=pretty)
diff --git a/src/somef/supervised_classification.py b/src/somef/supervised_classification.py
index 11da54b9..eac0ae05 100644
--- a/src/somef/supervised_classification.py
+++ b/src/somef/supervised_classification.py
@@ -1,3 +1,4 @@
+import logging
 import sys
 import pickle
 import pandas as pd
@@ -5,58 +6,68 @@
 from pathlib import Path
 from os import path
 from .rolf import preprocessing
+from .process_results import Result
 
 
-def run_category_classification(readme_text: str, threshold: float):
+def run_category_classification(readme_text: str, threshold: float, results: Result):
     """
     Function which returns the categories, confidence and technique of the given repo
     Parameters
     ----------
-    readme_text: the pure text of the readme
-    threshold: the threshold for the confidence
-
+    @param readme_text: the pure text of the readme
+    @param threshold: the threshold for the confidence
+    @param results: the JSON results to attach the resultant categories
     Returns
     -------
-    Returns the list of the results
+    @returns: A Result with the categories incorporated
     """
     df = pd.DataFrame([readme_text], columns=['Text'])
     preprocessing.Preprocessor(df).run()
     text = [df['Text'][0]]
-    res = []
-    for model_file in (Path(__file__).parent / 'rolf/models').iterdir():
-        with open(model_file, 'rb') as f:
-            model = pickle.load(f)
-            cat = model.predict(text).tolist()[0]
-            prob = max(model.predict_proba(text).tolist()[0])
-            if cat != 'Other' and prob > threshold:
-                res.append({constants.CONFIDENCE: [prob], constants.VALUE: [cat],
-                            constants.TECHNIQUE: constants.SUPERVISED_CLASSIFICATION})
-    return res
+    try:
+        for model_file in (Path(__file__).parent / 'rolf/models').iterdir():
+            with open(model_file, 'rb') as f:
+                model = pickle.load(f)
+                cat = model.predict(text).tolist()[0]
+                prob = max(model.predict_proba(text).tolist()[0])
+                if cat != 'Other' and prob > threshold:
+                    results.add_result(constants.CAT_APPLICATION_DOMAIN,
+                                       {
+                                           constants.PROP_TYPE: constants.STRING,
+                                           constants.PROP_VALUE: cat
+                                       }, prob, constants.TECHNIQUE_SUPERVISED_CLASSIFICATION)
+    except Exception as e:
+        logging.error("Error when applying supervised classification " + str(e))
+    return results
 
 
-def classify(scores, threshold, excerpts_headers, header_parents):
+def classify(scores, threshold, excerpts_headers, header_parents, repository_metadata: Result):
     """
     Function takes scores dictionary and a threshold as input
     Parameters
     ----------
-    scores: score dictionary passed as input
-    threshold: threshold to filter predictions (only predictions above threshold are returned)
-    excerpts_headers: headers to which each excerpt belongs (if any)
-    header_parents: parent headers of each excerpt
+    @param scores: score dictionary passed as input
+    @param threshold: threshold to filter predictions (only predictions above threshold are returned)
+    @param excerpts_headers: headers to which each excerpt belongs (if any)
+    @param header_parents: parent headers of each excerpt
+    @param repository_metadata: Result with the results of the repository so far
 
     Returns
     -------
-    Predictions containing excerpts with a confidence above the given threshold.
+    @returns Result including predictions and their text excerpts with a confidence above the given threshold.
     """
-    print("Checking Thresholds for Classified Excerpts.")
-    predictions = {}
+    logging.info("Checking thresholds for classified excerpts.")
+    try:
+        source = repository_metadata.results[constants.CAT_README_URL][0]
+        source = source[constants.PROP_RESULT][constants.PROP_VALUE]
+    except:
+        source = "README.md"
+    # print(scores)
     for ele in scores.keys():
-        print("Running for", ele)
-        flag = False
-        predictions[ele] = []
         excerpt = ""
-        confid = []
+        confidence = 0
         header = ""
+        num_elems = 1  # for calculating an average of confidence (in case multiple values are aggregated).
         for i in range(len(scores[ele]['confidence'])):
             if scores[ele]['confidence'][i] >= threshold:
                 element = scores[ele]['excerpt'][i]
@@ -67,7 +78,7 @@ def classify(scores, threshold, excerpts_headers, header_parents):
                         ind = elem.index.values[0]
                         header = elem.at[ind, 'header']
                     excerpt = excerpt + scores[ele]['excerpt'][i] + ' \n'
-                    confid.append(scores[ele]['confidence'][i])
+                    confidence = scores[ele]['confidence'][i]
                 else:
                     current_header = ""
                     if element in set(excerpts_headers['text']):
@@ -77,62 +88,35 @@ def classify(scores, threshold, excerpts_headers, header_parents):
                     # if both headers are the same, the new data is added
                     if header == current_header:
                         excerpt = excerpt + scores[ele]['excerpt'][i] + ' \n'
-                        confid.append(scores[ele]['confidence'][i])
-                    # if they are not the same, a new excerpt is created with the previous data
-                    # and stores the new data as part of a new excerpt
+                        confidence = confidence + scores[ele]['confidence'][i]
+                        num_elems = num_elems + 1
+                    # if they are not the same, add a result
                     else:
-                        if not header == "":
-                            element = remove_unimportant_excerpts(
-                                {'excerpt': excerpt, 'confidence': confid, 'originalHeader': header,
-                                 'parentHeader': header_parents[header]})
-                        else:
-                            element = remove_unimportant_excerpts({'excerpt': excerpt, 'confidence': confid})
-                        if len(element['confidence']) != 0:
-                            predictions[ele].append(element)
+                        result = {
+                            constants.PROP_TYPE: constants.TEXT_EXCERPT,
+                            constants.PROP_VALUE: excerpt
+                        }
+                        if header != "":
+                            result[constants.PROP_ORIGINAL_HEADER] = header
+                        repository_metadata.add_result(ele, result, confidence / num_elems,
+                                                       constants.TECHNIQUE_SUPERVISED_CLASSIFICATION, source)
                         header = current_header
+                        num_elems = 1
                         excerpt = scores[ele]['excerpt'][i] + ' \n'
-                        confid = [scores[ele]['confidence'][i]]
+                        confidence = scores[ele]['confidence'][i]
         # if an element hasn't been added, it's added at this point
         if excerpt != "":
-            if not header == "":
-                element = remove_unimportant_excerpts(
-                    {'excerpt': excerpt, 'confidence': confid, 'originalHeader': header,
-                     'parentHeader': header_parents[header]})
-            else:
-                element = remove_unimportant_excerpts({'excerpt': excerpt, 'confidence': confid})
-            if len(element['confidence']) != 0:
-                predictions[ele].append(element)
-        print("Run completed.")
-    print("All excerpts below the given threshold have been removed. \n")
-    return predictions
-
-
-def remove_unimportant_excerpts(excerpt_element):
-    """
-    Function which removes all excerpt lines which have been classified but contain only one word.
-    TO DO: It does not seem to filter lines with one word
-    Parameters
-    ----------
-    excerpt_element: excerpt to process
-
-    Returns
-    -------
-    Returns the excerpt to be entered into the predictions
-    """
-    excerpt_info = excerpt_element['excerpt']
-    excerpt_confidence = excerpt_element[constants.CONFIDENCE]
-    if 'originalHeader' in excerpt_element:
-        final_excerpt = {'excerpt': "", 'confidence': [], constants.TECHNIQUE: constants.SUPERVISED_CLASSIFICATION,
-                         'originalHeader': ""}
-    else:
-        final_excerpt = {'excerpt': "", 'confidence': [], constants.TECHNIQUE: constants.SUPERVISED_CLASSIFICATION}
-    final_excerpt['excerpt'] += excerpt_info
-    final_excerpt['confidence'] = excerpt_confidence
-    if 'originalHeader' in excerpt_element:
-        final_excerpt['originalHeader'] += excerpt_element['originalHeader']
-    if 'parentHeader' in excerpt_element and excerpt_element['parentHeader'] != "":
-        final_excerpt['parentHeader'] = excerpt_element['parentHeader']
-    return final_excerpt
+            result = {
+                constants.PROP_TYPE: constants.TEXT_EXCERPT,
+                constants.PROP_VALUE: excerpt
+            }
+            if header != "":
+                result[constants.PROP_ORIGINAL_HEADER] = header
+            repository_metadata.add_result(ele, result, confidence / num_elems,
+                                           constants.TECHNIQUE_SUPERVISED_CLASSIFICATION,
+                                           source)
+    logging.info("All excerpts below the threshold have been removed. \n")
+    return repository_metadata
 
 
 def run_classifiers(excerpts, file_paths):
@@ -150,23 +134,25 @@ def run_classifiers(excerpts, file_paths):
 
     """
     score_dict = {}
-    if len(excerpts) > 0:
-        text_to_classifier = []
-        text_to_results = []
-        for key in excerpts.keys():
-            text_to_classifier.append(key)
-            text_to_results.append(excerpts[key])
-        for category in constants.categories:
-            if category not in file_paths.keys():
-                sys.exit("Error: Category " + category + " file path not present in config.json")
-            file_name = file_paths[category]
-            if not path.exists(file_name):
-                sys.exit(f"Error: File or Directory {file_name} does not exist")
-            print("Classifying excerpts for the category", category)
-            classifier = pickle.load(open(file_name, 'rb'))
-            scores = classifier.predict_proba(text_to_classifier)
-            score_dict[category] = {'excerpt': text_to_results, 'confidence': scores[:, 1]}
-            print("Excerpt Classification Successful for the Category", category)
-        print("\n")
+    try:
+        if len(excerpts) > 0:
+            text_to_classifier = []
+            text_to_results = []
+            for key in excerpts.keys():
+                text_to_classifier.append(key)
+                text_to_results.append(excerpts[key])
+            for category in constants.supervised_categories:
+                if category not in file_paths.keys():
+                    sys.exit("Error: Category " + category + " file path not present in config.json")
+                file_name = file_paths[category]
+                if not path.exists(file_name):
+                    sys.exit(f"Error: File or Directory {file_name} does not exist")
+                logging.info("Classifying excerpts for the category " + category)
+                classifier = pickle.load(open(file_name, 'rb'))
+                scores = classifier.predict_proba(text_to_classifier)
+                score_dict[category] = {'excerpt': text_to_results, 'confidence': scores[:, 1]}
+                # logging.info("Excerpt classification successful category"+ category)
+    except Exception as e:
+        logging.error("Error while running supervised classifiers on README " + str(e))
 
     return score_dict
diff --git a/src/somef/utils/constants.py b/src/somef/utils/constants.py
index 2154cfa5..dc7f3746 100644
--- a/src/somef/utils/constants.py
+++ b/src/somef/utils/constants.py
@@ -17,70 +17,203 @@
 REGEXP_REDDIT = "(https://www.reddit.com/r/"
 REGEXP_DISCORD = "(https://discord.com/invite/"
 REGEXP_GITTER = "[![Gitter chat]"
+# These are two of the most common ones for Python, but it may be expanded
 REGEXP_PYPI = "[![PyPI]"
+REGEXP_PYPI_2 = "[![Latest PyPI version]"
 REGEXP_COLAB = "https://colab.research.google.com/drive"
 REGEXP_BIBTEX = r'\@[a-zA-Z]+\{[.\n\S\s]+?[author|title][.\n\S\s]+?[author|title][.\n\S\s]+?\n\}'
 REGEXP_DOI = r'\[\!\[DOI\]([^\]]+)\]\(([^)]+)\)'
 REGEXP_LINKS = r"\[(.*?)?\]\(([^)]+)\)"
 REGEXP_IMAGES = r"!\[(.*?)?\]\((.*?)?\)"
-                #r"!\[[^\]]*\]\((.*?)?\)"
+# r"!\[[^\]]*\]\((.*?)?\)"
 
 
-categories = ['description', 'citation', 'installation', 'invocation']
+# Categories recognized by SOMEF (they all start by CAT_
+CAT_APPLICATION_DOMAIN = "application_domain"
+CAT_ACKNOWLEDGEMENT = "acknowledgement"
+CAT_CITATION = "citation"
+CAT_CONTRIBUTORS = "contributors"
+CAT_CONTRIBUTING_GUIDELINES = "contributing_guidelines"
+CAT_COC = "code_of_conduct"
+CAT_CODE_REPOSITORY = "code_repository"
+CAT_CONTACT = "contact"
+CAT_DATE_CREATED = "date_created"
+CAT_DATE_UPDATED = "date_updated"
+CAT_DESCRIPTION = "description"
+CAT_DOCUMENTATION = "documentation"
+CAT_DOWNLOAD = "download"
+CAT_DOWNLOAD_URL = "download_url"
+CAT_EXECUTABLE_EXAMPLE = "executable_example"
+CAT_FAQ = "faq"
+CAT_FORK_COUNTS = "forks_count"
+CAT_FORKS_URLS = "forks_url"
+CAT_FULL_NAME = "full_name"
+CAT_FULL_TITLE = "full_title"
+CAT_HAS_BUILD_FILE = "has_build_file"
+CAT_HAS_EXECUTABLE_NOTEBOOK = "has_executable_notebook"
+CAT_HAS_SCRIPT_FILE = "has_script_file"
+CAT_IDENTIFIER = "identifier"
+CAT_IMAGE = "image"
+CAT_INSTALLATION = "installation"
+CAT_INVOCATION = "invocation"
+CAT_ISSUE_TRACKER = "issue_tracker"
+CAT_KEYWORDS = "keywords"
+CAT_LICENSE = "license"
+CAT_LOGO = "logo"
+CAT_NAME = "name"
+CAT_ONTOLOGIES = "ontologies"
+CAT_OWNER = "owner"
+CAT_PACKAGE_DISTRIBUTION = "package_distribution"
+CAT_PROGRAMMING_LANGUAGES = "programming_languages"
+CAT_README_URL = "readme_url"
+CAT_RELATED_DOCUMENTATION = "related_documentation"
+CAT_RELEASES = "releases"
+CAT_RUN = "run"
+CAT_STATUS = "repository_status"
+CAT_REQUIREMENTS = "requirements"
+CAT_STARS = "stargazers_count"
+CAT_SUPPORT = "support"
+CAT_SUPPORT_CHANNELS = "support_channels"
+CAT_USAGE = "usage"
 
-# Constants for labeling the output JSON
-VALUE = "value"
-TECHNIQUE = "technique"
-CONFIDENCE = "confidence"
-# Different technique types
-SUPERVISED_CLASSIFICATION = "supervised classification"
-HEADER_ANALYSIS = "header analysis"
-REGULAR_EXPRESSION = "regular expression"
-FILE_EXPLORATION = "file exploration"
+# Special category: missing categories
+CAT_MISSING = "somef_missing_categories"
 
-# keep_keys = ('description', 'name', 'owner', 'license', 'languages_url', 'forks_url')
-# instead of keep keys, we have this table
-# it says that we want the key "codeRepository", and that we'll get it from the path "html_url" within the result object
+# list of those categories to be analyzed with supervised classification.
+# supervised_categories = [CAT_DESCRIPTION, CAT_CITATION, CAT_INSTALLATION, CAT_INVOCATION]
+supervised_categories = [CAT_DESCRIPTION, CAT_INSTALLATION, CAT_INVOCATION]
 
+# list with all categories
+all_categories = [CAT_APPLICATION_DOMAIN, CAT_ACKNOWLEDGEMENT, CAT_CITATION, CAT_CONTRIBUTORS,
+                  CAT_CONTRIBUTING_GUIDELINES,
+                  CAT_COC, CAT_CODE_REPOSITORY, CAT_CONTACT, CAT_DESCRIPTION, CAT_DATE_CREATED, CAT_DATE_UPDATED,
+                  CAT_DOCUMENTATION, CAT_DOWNLOAD, CAT_DOWNLOAD_URL, CAT_EXECUTABLE_EXAMPLE,
+                  CAT_FAQ, CAT_FORK_COUNTS, CAT_FORKS_URLS, CAT_FULL_NAME, CAT_FULL_TITLE, CAT_HAS_BUILD_FILE,
+                  CAT_HAS_EXECUTABLE_NOTEBOOK, CAT_HAS_SCRIPT_FILE, CAT_IDENTIFIER, CAT_IMAGE, CAT_INSTALLATION,
+                  CAT_INVOCATION, CAT_ISSUE_TRACKER, CAT_KEYWORDS, CAT_LICENSE, CAT_LOGO, CAT_NAME, CAT_ONTOLOGIES,
+                  CAT_OWNER, CAT_PACKAGE_DISTRIBUTION, CAT_PROGRAMMING_LANGUAGES, CAT_README_URL,
+                  CAT_RELATED_DOCUMENTATION, CAT_RELEASES, CAT_RUN,
+                  CAT_STATUS, CAT_REQUIREMENTS, CAT_STARS, CAT_SUPPORT, CAT_SUPPORT_CHANNELS, CAT_USAGE]
+
+# All properties used by SOMEF to label the output JSON
+# Provenance:
+PROP_PROVENANCE = "somef_provenance"
+PROP_SOMEF_VERSION = "somef_version"
+PROP_SOMEF_SCHEMA_VERSION = "somef_schema_version"
+PROP_DATE = "date"
+# for Category:
+PROP_CONFIDENCE = "confidence"
+PROP_RESULT = "result"
+PROP_SOURCE = "source"
+PROP_TECHNIQUE = "technique"
+# for Result:
+PROP_FORMAT = "format"
+PROP_TYPE = "type"
+AGENT_TYPE = "agent_type"  # Special type needed when objects are nested
+PROP_VALUE = "value"
+# For Result types
+PROP_AUTHOR = "author"
+PROP_DOI = "doi"
+PROP_DESCRIPTION = "description"
+PROP_DATE_CREATED = "date_created"
+PROP_DATE_PUBLISHED = "date_published"
+PROP_DATE_UPDATED = "date_updated"
+PROP_HTML_URL = "html_url"
+PROP_NAME = "name"
+PROP_ORIGINAL_HEADER = "original_header"
+PROP_PARENT_HEADER = "parent_header"
+PROP_SIZE = "size"
+PROP_SPDX_ID = "spdx_id"
+PROP_TAG = "tag"
+PROP_URL = "url"
+PROP_ZIPBALL_URL = "zipball_url"
+PROP_TARBALL_URL = "tarball_url"
+
+# Format:
+FORMAT_BIB = "bibtex"
+FORMAT_CFF = "cff"
+FORMAT_JUPYTER_NB = "jupyter_notebook"
+FORMAT_DOCKERFILE = "dockerfile"
+FORMAT_DOCKER_COMPOSE = "docker_compose"
+FORMAT_READTHEDOCS = "readthedocs"
+FORMAT_WIKI = "wiki"
+
+# Result types: data types
+STRING = "String"
+URL = "Url"
+DATE = "Date"
+NUMBER = "Number"
+TEXT_EXCERPT = "Text_excerpt"
+FILE_DUMP = "File_dump"
+
+# Result types: entities (complex objects)
+AGENT = "Agent"
+RELEASE = "Release"
+LICENSE = "License"
+PUBLICATION = "Publication"
+LANGUAGE = "Programming_language"
+
+# Different techniques
+TECHNIQUE_SUPERVISED_CLASSIFICATION = "supervised_classification"
+TECHNIQUE_HEADER_ANALYSIS = "header_analysis"
+TECHNIQUE_REGULAR_EXPRESSION = "regular_expression"
+TECHNIQUE_FILE_EXPLORATION = "file_exploration"
+TECHNIQUE_CODE_CONFIG_PARSER = "code_parser"
+TECHNIQUE_GITHUB_API = "GitHub_API"
+TECHNIQUE_GITLAB_API = "GitLab_API"
+
+# GitHub properties
+GITHUB_DOMAIN = "github.com"
+GITHUB_ACCEPT_HEADER = "application/vnd.github.v3+json"
+GITHUB_API = "https://api.github.com/repos"
+
+# GitLab metadata fields
+
+
+# Crosswalk to retrieve easily contents of interest from the GitHub response
 github_crosswalk_table = {
-    "codeRepository": "html_url",
+    CAT_CODE_REPOSITORY: "html_url",
     "languages_url": "languages_url",
-    "owner": ["owner", "login"],
-    "ownerType": ["owner", "type"],  # used to determine if owner is User or Organization
-    "dateCreated": "created_at",
-    "dateModified": "updated_at",
-    "license": "license",
-    "description": "description",
-    "name": "name",
-    "fullName": "full_name",
-    "issueTracker": "issues_url",
-    "forksUrl": "forks_url",
-    "stargazers_count": "stargazers_count",
-    "forks_count": "forks_count"
+    CAT_OWNER: ["owner", "login"],
+    AGENT_TYPE: ["owner", "type"],  # used to determine if owner is User or Organization
+    CAT_DATE_CREATED: "created_at",
+    CAT_DATE_UPDATED: "updated_at",
+    CAT_LICENSE: "license",
+    CAT_DESCRIPTION: "description",
+    CAT_NAME: "name",
+    CAT_FULL_NAME: "full_name",
+    CAT_ISSUE_TRACKER: "issues_url",
+    CAT_FORKS_URLS: "forks_url",
+    CAT_STARS: "stargazers_count",
+    CAT_KEYWORDS: "topics",
+    CAT_FORK_COUNTS: "forks_count"
 }
 
+# Mapping for
 release_crosswalk_table = {
-    'tagName': 'tag_name',
-    'name': 'name',
-    'authorName': ['author', 'login'],
-    'authorType': ['author', 'type'],
-    'body': 'body',
-    'tarballUrl': 'tarball_url',
-    'zipballUrl': 'zipball_url',
-    'htmlUrl': 'html_url',
-    'url': 'url',
-    'dateCreated': 'created_at',
-    'datePublished': "published_at",
+    PROP_TAG: 'tag_name',
+    PROP_NAME: 'name',
+    PROP_AUTHOR: ['author', 'login'],
+    AGENT_TYPE: ['author', 'type'],
+    PROP_DESCRIPTION: 'body',
+    PROP_TARBALL_URL: 'tarball_url',
+    PROP_ZIPBALL_URL: 'zipball_url',
+    PROP_HTML_URL: 'html_url',
+    PROP_URL: 'url',
+    PROP_DATE_CREATED: 'created_at',
+    PROP_DATE_PUBLISHED: "published_at",
 }
 
-categories_files_header = ["installation", "citation", "acknowledgement", "run", "download", "requirement", "contact",
-            "description", "contributor", "documentation", "license", "usage", "faq", "support", "identifier",
-              "hasExecutableNotebook", "hasBuildFile", "hasDocumentation", "executableExample"]
-
+# TO DO: Assess run and download.
+categories_files_header = [CAT_INSTALLATION, CAT_CITATION, CAT_ACKNOWLEDGEMENT, "run", "download", CAT_REQUIREMENTS,
+                           CAT_CONTACT, CAT_DESCRIPTION, CAT_CONTRIBUTORS, CAT_DOCUMENTATION, CAT_LICENSE, CAT_USAGE,
+                           CAT_FAQ, CAT_SUPPORT, CAT_IDENTIFIER, CAT_HAS_EXECUTABLE_NOTEBOOK, CAT_HAS_BUILD_FILE,
+                           CAT_EXECUTABLE_EXAMPLE]
 
+# This will not be needed anymore. REMOVE
 file_exploration = ['hasExecutableNotebook', 'hasBuildFile', 'hasDocumentation', 'codeOfConduct',
-                        'contributingGuidelines', 'licenseFile', 'licenseText', 'acknowledgement',
-                        'contributors', 'hasScriptFile', 'ontologies']
+                    'contributingGuidelines', 'licenseFile', 'licenseText', 'acknowledgement',
+                    'contributors', 'hasScriptFile', 'ontologies']
 
 
 class RepositoryType(Enum):
diff --git a/src/somef/utils/markdown_utils.py b/src/somef/utils/markdown_utils.py
index 8f4d0325..d2cfad53 100644
--- a/src/somef/utils/markdown_utils.py
+++ b/src/somef/utils/markdown_utils.py
@@ -1,6 +1,9 @@
+import logging
 from io import StringIO
 from markdown import Markdown
-from .. import regular_expressions
+from ..utils import constants
+import re
+
 
 ## Markdown to plain text conversion: begin ##
 # code snippet from https://stackoverflow.com/a/54923798
@@ -38,12 +41,12 @@ def remove_bibtex(string_list):
         The strings list without bibtex blocks
         """
     for x, element in enumerate(string_list):
-        bib_references = regular_expressions.extract_bibtex(element)
+        bib_references = re.findall(constants.REGEXP_BIBTEX, element)
         if len(bib_references) > 0:
             top = element.find(bib_references[0])
             init = element.rfind("```", 0, top)
             end = element.find("```", init + 3)
             substring = element[init:end + 3]
             string_list[x] = element.replace(substring, "")
-    print("Extraction of bibtex citation from readme completed. \n")
+    logging.info("Extraction of bibtex citation from readme completed. \n")
     return string_list


Test patch:
diff --git a/src/somef/rolf/models/astrophysics.sav b/src/somef/rolf/models_test/astrophysics.sav
similarity index 100%
rename from src/somef/rolf/models/astrophysics.sav
rename to src/somef/rolf/models_test/astrophysics.sav
diff --git a/src/somef/rolf/models_test/rationale.md b/src/somef/rolf/models_test/rationale.md
new file mode 100644
index 00000000..bc96b495
--- /dev/null
+++ b/src/somef/rolf/models_test/rationale.md
@@ -0,0 +1,5 @@
+This folder has models that are in testing process.
+
+The astrophysics domain produces nice results, but it needs revision with negative samples.
+
+The sequential domain has too few labels, hence producing results that can be improved
\ No newline at end of file
diff --git a/src/somef/rolf/models/sequential.sav b/src/somef/rolf/models_test/sequential.sav
similarity index 100%
rename from src/somef/rolf/models/sequential.sav
rename to src/somef/rolf/models_test/sequential.sav
diff --git a/src/somef/test/test_cli.py b/src/somef/test/test_cli.py
index 79c7348c..309dbd92 100644
--- a/src/somef/test/test_cli.py
+++ b/src/somef/test/test_cli.py
@@ -4,6 +4,7 @@
 import validators
 from pathlib import Path
 from .. import somef_cli
+from ..utils import constants
 
 test_data_path = str(Path(__file__).parent / "test_data") + os.path.sep
 test_data_repositories = str(Path(__file__).parent / "test_data" / "repositories") + os.path.sep
@@ -79,7 +80,7 @@ def test_issue_281(self):
         text_file = open(test_data_path + "test-281.json", "r")
         data = text_file.read()
         text_file.close()
-        assert data.find("missingCategories") > 0
+        assert data.find(constants.CAT_MISSING) > 0
         os.remove(test_data_path + "test-281.json")
 
     def test_issue_200(self):
@@ -98,7 +99,7 @@ def test_issue_200(self):
         text_file = open(test_data_path + "test-200.json", "r")
         data = text_file.read()
         text_file.close()
-        assert data.find("parentHeader") > 0
+        assert data.find(constants.PROP_PARENT_HEADER) > 0
         os.remove(test_data_path + "test-200.json")
 
     def test_issue_343(self):
@@ -117,7 +118,7 @@ def test_issue_343(self):
         text_file = open(test_data_path + "test-343.json", "r")
         data = text_file.read()
         text_file.close()
-        assert data.find("parentHeader") > 0
+        assert data.find(constants.PROP_PARENT_HEADER) > 0
         os.remove(test_data_path + "test-343.json")
 
     def test_issue_355(self):
@@ -136,9 +137,8 @@ def test_issue_355(self):
                           missing=False)
         text_file = open(test_data_path + "repositories/repos_oeg/test-355.json", "r")
         data = text_file.read()
-        print(data)
         text_file.close()
-        assert data.find("longTitle") > 0
+        assert data.find(constants.CAT_FULL_TITLE) > 0
         os.remove(test_data_path + "repositories/repos_oeg/test-355.json")
 
     def test_issue_150(self):
@@ -158,7 +158,7 @@ def test_issue_150(self):
         text_file = open(test_data_path + "test-150.json-ld", "r")
         data = text_file.read()
         text_file.close()
-        assert data.find("acknowledgement") == -1
+        assert data.find(constants.CAT_ACKNOWLEDGEMENT) == -1
         os.remove(test_data_path + "test-150.json-ld")
 
     def test_issue_346(self):
@@ -196,7 +196,7 @@ def test_issue_241(self):
         text_file = open(test_data_path + "test-241.json", "r")
         data = text_file.read()
         text_file.close()
-        assert data.find("packageDistribution") > 0
+        assert data.find(constants.CAT_PACKAGE_DISTRIBUTION) > 0
         os.remove(test_data_path + "test-241.json")
 
     def test_issue_380(self):
@@ -215,7 +215,7 @@ def test_issue_380(self):
         text_file = open(test_data_path + "test-380.json", "r")
         data = text_file.read()
         text_file.close()
-        assert data.find("installation") > 0
+        assert data.find(constants.CAT_INSTALLATION) > 0
         os.remove(test_data_path + "test-380.json")
 
     def test_issue_379(self):
@@ -235,7 +235,7 @@ def test_issue_379(self):
         data = text_file.read()
         text_file.close()
         json_content = json.loads(data)
-        description = json_content['description']
+        description = json_content[constants.CAT_DESCRIPTION]
         assert len(description) > 0
         os.remove(test_data_path + "test-379.json")
 
@@ -279,7 +279,7 @@ def test_issue_384(self):
 
     def test_issue_378(self):
         """Checks classification of a header based on what was written in upper level headers"""
-        somef_cli.run_cli(threshold=0.8,
+        somef_cli.run_cli(threshold=0.9,
                           ignore_classifiers=False,
                           repo_url=None,
                           doc_src=test_data_path + "README-easytv-annotator.md",
@@ -294,11 +294,14 @@ def test_issue_378(self):
         data = text_file.read()
         text_file.close()
         json_content = json.loads(data)
-        installation = json_content['installation']
+        installation = json_content[constants.CAT_INSTALLATION]
         assert_true = False
+        print(installation)
         for install in installation:
-            if 'parentHeader' in install.keys():
-                if install['originalHeader'] == "Hard way" and "Installation" in install['parentHeader']:
+            result_install = install[constants.PROP_RESULT]
+            if constants.PROP_PARENT_HEADER in result_install.keys():
+                if result_install[constants.PROP_ORIGINAL_HEADER] == "Hard way" and \
+                        "Installation" in result_install[constants.PROP_PARENT_HEADER]:
                     assert_true = True
         assert assert_true
         os.remove(test_data_path + "test-378.json")
@@ -320,15 +323,17 @@ def test_issue_378_2(self):
         data = text_file.read()
         text_file.close()
         json_content = json.loads(data)
-        installation = json_content['installation']
-        assert_true = False
+        installation = json_content[constants.CAT_INSTALLATION]
+        final_result = False
         for install in installation:
-            if 'parentHeader' in install.keys():
-                if install['originalHeader'] == "Easy way" and "Installation" in install['parentHeader'] \
-                        and "easytv-annotator" in install["parentHeader"]:
-                    assert_true = True
-        assert assert_true
+            install_result = install[constants.PROP_RESULT]
+            if constants.PROP_PARENT_HEADER in install_result.keys():
+                if install_result[constants.PROP_ORIGINAL_HEADER] == "Easy way" and \
+                        "Installation" in install_result[constants.PROP_PARENT_HEADER] \
+                        and "easytv-annotator" in install_result[constants.PROP_PARENT_HEADER]:
+                    final_result = True
         os.remove(test_data_path + "test-378-2.json")
+        assert final_result
 
     def test_issue_260(self):
         """Checks if colab notebooks are detected"""
@@ -346,7 +351,7 @@ def test_issue_260(self):
         text_file = open(test_data_path + "test-260.json", "r")
         data = text_file.read()
         text_file.close()
-        assert data.find("executableExample") > 0
+        assert data.find(constants.CAT_EXECUTABLE_EXAMPLE) > 0
         os.remove(test_data_path + "test-260.json")
 
     def test_issue_319_1(self):
@@ -366,7 +371,7 @@ def test_issue_319_1(self):
         text_file = open(test_data_path + "test-319-1.json", "r")
         data = text_file.read()
         text_file.close()
-        assert data.find("citation file format") > 0
+        assert data.find(constants.FORMAT_CFF) > 0
         os.remove(test_data_path + "test-319-1.json")
 
     def test_issue_388(self):
@@ -387,7 +392,7 @@ def test_issue_388(self):
         data = text_file.read()
         json_content = json.loads(data)
         text_file.close()
-        assert len(json_content["ontologies"]["excerpt"]) == 2
+        assert len(json_content[constants.CAT_ONTOLOGIES]) == 2
         os.remove(test_data_path + "test-388.json")
 
     def test_issue_319_2(self):
@@ -407,7 +412,7 @@ def test_issue_319_2(self):
         text_file = open(test_data_path + "test-319-2.json", "r")
         data = text_file.read()
         text_file.close()
-        assert data.find("jupyter notebook") > 0
+        assert data.find(constants.FORMAT_JUPYTER_NB) > 0
         os.remove(test_data_path + "test-319-2.json")
 
     def test_issue_385(self):
@@ -427,7 +432,7 @@ def test_issue_385(self):
         data = text_file.read()
         text_file.close()
         json_content = json.loads(data)
-        usage = json_content['usage']
+        usage = json_content[constants.CAT_USAGE]
         assert len(usage) == 3
         os.remove(test_data_path + "test-385.json")
 
@@ -435,7 +440,8 @@ def test_issue_398(self):
         """Checks that repostatus has confidence (always 1.0)"""
         somef_cli.run_cli(threshold=0.8,
                           ignore_classifiers=False,
-                          repo_url="https://github.com/dgarijo/Widoco",
+                          repo_url=None,
+                          local_repo=test_data_repositories + "Widoco",
                           doc_src=None,
                           in_file=None,
                           output=test_data_path + "test-398.json",
@@ -448,9 +454,9 @@ def test_issue_398(self):
         data = text_file.read()
         text_file.close()
         json_content = json.loads(data)
-        repo_status = json_content['repoStatus']
-        confidence = repo_status['confidence']
-        assert len(confidence) == 1
+        repo_status = json_content[constants.CAT_STATUS][0]
+        confidence = repo_status[constants.PROP_CONFIDENCE]
+        assert confidence == 1
         os.remove(test_data_path + "test-398.json")
 
     # def test_issue_393(self):
@@ -492,7 +498,7 @@ def test_issue_314(self):
         data = text_file.read()
         text_file.close()
         json_content = json.loads(data)
-        acknowledgement = json_content['acknowledgement']
+        acknowledgement = json_content[constants.CAT_ACKNOWLEDGEMENT]
         assert acknowledgement is not None
         os.remove(test_data_path + "test-314.json")
 
@@ -514,7 +520,7 @@ def test_issue_314_1(self):
         data = text_file.read()
         text_file.close()
         json_content = json.loads(data)
-        image = json_content['image']
+        image = json_content[constants.CAT_IMAGE]
         assert image is not None
         os.remove(test_data_path + "test-314-1.json")
 
@@ -536,7 +542,7 @@ def test_issue_314_2(self):
         data = text_file.read()
         text_file.close()
         json_content = json.loads(data)
-        description = json_content['description']
+        description = json_content[constants.CAT_DESCRIPTION]
         assert description is not None
         os.remove(test_data_path + "test-314-2.json")
 
@@ -558,7 +564,7 @@ def test_issue_314_3(self):
         data = text_file.read()
         text_file.close()
         json_content = json.loads(data)
-        description = json_content['description']
+        description = json_content[constants.CAT_DESCRIPTION]
         assert description is not None
         os.remove(test_data_path + "test-314-3.json")
 
@@ -573,12 +579,12 @@ def test_gitlab(self):
         data = text_file.read()
         text_file.close()
         json_content = json.loads(data)
-        description = json_content['description']
+        description = json_content[constants.CAT_DESCRIPTION]
         assert description is not None
         os.remove(test_data_path + "test-314-2.json")
 
     def test_issue_403(self):
-        """Checks that the readme returned by somef is correct"""
+        """Checks that the readme link returned by somef is correct"""
         somef_cli.run_cli(threshold=0.8,
                           ignore_classifiers=False,
                           repo_url="https://github.com/oeg-upm/wot-hive",
@@ -595,13 +601,14 @@ def test_issue_403(self):
         data = text_file.read()
         text_file.close()
         json_content = json.loads(data)
-        readme_url = json_content['readmeUrl']
-        excerpt = readme_url['excerpt']
+        readme_url = json_content[constants.CAT_README_URL][0]
+        excerpt = readme_url[constants.PROP_RESULT][constants.PROP_VALUE]
         assert excerpt == 'https://raw.githubusercontent.com/oeg-upm/wot-hive/main/README.md'
         os.remove(test_data_path + "test-403.json")
 
     def test_issue_408(self):
-        """Checks that the documentation links (docs) are properly found"""
+        """Checks that the documentation links (docs) are found only if the docs folder has documents that seem like
+        documentation"""
         somef_cli.run_cli(threshold=0.8,
                           ignore_classifiers=False,
                           repo_url=None,
@@ -622,28 +629,6 @@ def test_issue_408(self):
             "https://github.com/oeg-upm/bimerr-epw/tree/master/Code/TDATA2RDFANDV/static/rest_framework/docs") == -1
         os.remove(test_data_path + "test-408.json")
 
-    def test_issue_408_1(self):
-        """Checks that the documentation links (docs) are properly found"""
-        somef_cli.run_cli(threshold=0.8,
-                          ignore_classifiers=False,
-                          repo_url="https://github.com/oeg-upm/bimerr-epw/",
-                          local_repo=None,
-                          doc_src=None,
-                          in_file=None,
-                          output=test_data_path + "test-408-1.json",
-                          graph_out=None,
-                          graph_format="turtle",
-                          codemeta_out=None,
-                          pretty=True,
-                          missing=True,
-                          readme_only=False)
-        text_file = open(test_data_path + "test-408-1.json", "r")
-        data = text_file.read()
-        text_file.close()
-        assert data.find(
-            "https://github.com/oeg-upm/bimerr-epw/tree/master/Code/TDATA2RDFANDV/static/rest_framework/docs") == -1
-        os.remove(test_data_path + "test-408-1.json")
-
     def test_issue_225_406(self):
         """Checks if wiki links are detected"""
         somef_cli.run_cli(threshold=0.8,
@@ -662,7 +647,8 @@ def test_issue_225_406(self):
         text_file = open(test_data_path + "test-225.json", "r")
         data = text_file.read()
         text_file.close()
-        assert data.find("\"type\": \"wiki\"")
+        print(data)
+        assert data.find("\""+constants.PROP_FORMAT+"\": \""+constants.FORMAT_WIKI+"\"")
         os.remove(test_data_path + "test-225.json")
 
     def test_issue_406(self):
@@ -683,7 +669,7 @@ def test_issue_406(self):
         text_file = open(test_data_path + "test-406.json", "r")
         data = text_file.read()
         text_file.close()
-        assert data.find("\"type\": \"readthedocs\"")
+        assert data.find("\""+constants.PROP_FORMAT+"\": \""+constants.FORMAT_READTHEDOCS+"\"") >= 0
         os.remove(test_data_path + "test-406.json")
 
     def test_issue_255(self):
@@ -708,7 +694,7 @@ def test_issue_255(self):
         os.remove(test_data_path + "test-255.json")
 
     def test_issue_255_1(self):
-        """Tests if somef can detect wiki articles"""
+        """Tests if somef can detect the abscence of wikis if a repo does not have it"""
         somef_cli.run_cli(threshold=0.8,
                           ignore_classifiers=False,
                           repo_url="https://github.com/PyLops/pylops/",
@@ -809,7 +795,7 @@ def test_issue_353(self):
         text_file = open(test_data_path + "test-353.json", "r")
         data = text_file.read()
         text_file.close()
-        assert data.find("description") >= 0
+        assert data.find(constants.CAT_DESCRIPTION) >= 0
         os.remove(test_data_path + "test-353.json")
 
     def test_issue_366(self):
@@ -830,11 +816,11 @@ def test_issue_366(self):
         text_file = open(test_data_path + "test-366.json", "r")
         data = text_file.read()
         text_file.close()
-        assert data.find("Docker file") >= 0
+        assert data.find(constants.FORMAT_DOCKERFILE) >= 0
         os.remove(test_data_path + "test-366.json")
 
     def test_issue_417(self):
-        """Checks for different codemeta errors"""
+        """Checks whether a repository correctly extracts to Codemeta"""
         somef_cli.run_cli(threshold=0.8,
                           ignore_classifiers=False,
                           repo_url="https://github.com/dgarijo/Widoco",
@@ -852,35 +838,18 @@ def test_issue_417(self):
         data = text_file.read()
         text_file.close()
         json_content = json.loads(data)
-        issue_tracker = json_content['issueTracker']
-        assert issue_tracker == 'https://api.github.com/repos/dgarijo/Widoco/issues'
+        issue_tracker = json_content["issueTracker"] # JSON is in Codemeta format
+        assert issue_tracker == 'https://github.com/dgarijo/Widoco/issues' and len(json_content["citation"])>0  and \
+               len(json_content["name"]) > 0 and len(json_content["identifier"])>0 and \
+               len(json_content["description"])>0 and len(json_content["readme"])>0 and \
+               len(json_content["author"])>0 and len(json_content["buildInstructions"])>0 and \
+               len(json_content["softwareRequirements"]) > 0 and len(json_content["programmingLanguage"]) > 0 and \
+               len(json_content["keywords"]) > 0 and len(json_content["logo"]) > 0 and \
+               len(json_content["license"]) > 0 and len(json_content["dateCreated"]) > 0
         os.remove(test_data_path + "test-417.json-ld")
 
-    def test_issue_382(self):
-        """Checks a given github repo works fine (contact section)"""
-        somef_cli.run_cli(threshold=0.8,
-                          ignore_classifiers=False,
-                          repo_url="https://github.com/oeg-upm/mapeathor",
-                          local_repo=None,
-                          doc_src=None,
-                          in_file=None,
-                          output=test_data_path + "test-382.json",
-                          graph_out=None,
-                          graph_format="turtle",
-                          codemeta_out=None,
-                          pretty=True,
-                          missing=True,
-                          readme_only=False)
-        text_file = open(test_data_path + "test-382.json", "r")
-        data = text_file.read()
-        text_file.close()
-        json_content = json.loads(data)
-        contact = json_content['contact']
-        assert contact is not None
-        os.remove(test_data_path + "test-382.json")
-
     def test_issue_428(self):
-        """Checks a given github repo works fine (contact section)"""
+        """Checks if the text before the main header is passed on to the classifiers"""
         somef_cli.run_cli(threshold=0.8,
                           ignore_classifiers=False,
                           repo_url=None,
@@ -898,7 +867,7 @@ def test_issue_428(self):
         data = text_file.read()
         text_file.close()
         assert data.find(
-            "Manim is an engine for precise programmatic animations, designed for creating explanatory math videos.") > 0
+           "Manim is an engine for precise programmatic animations, designed for creating explanatory math videos.") > 0
         os.remove(test_data_path + "test-428.json")
 
     def test_issue_443(self):
@@ -919,29 +888,7 @@ def test_issue_443(self):
         data = text_file.read()
         text_file.close()
         json_content = json.loads(data)
-        installation = json_content['installation']
-        assert installation is not None
-        os.remove(test_data_path + "test-443.json")
-
-    def test_issue_443_2(self):
-        somef_cli.run_cli(threshold=0.8,
-                          ignore_classifiers=False,
-                          repo_url="https://github.com/oeg-upm/awesome-semantic-web",
-                          local_repo=None,
-                          doc_src=None,
-                          in_file=None,
-                          output=test_data_path + "test-443.json",
-                          graph_out=None,
-                          graph_format="turtle",
-                          codemeta_out=None,
-                          pretty=True,
-                          missing=True,
-                          readme_only=False)
-        text_file = open(test_data_path + "test-443.json", "r")
-        data = text_file.read()
-        text_file.close()
-        json_content = json.loads(data)
-        installation = json_content['installation']
+        installation = json_content[constants.CAT_INSTALLATION]
         assert installation is not None
         os.remove(test_data_path + "test-443.json")
 
@@ -963,7 +910,7 @@ def test_issue_443_3(self):
         data = text_file.read()
         text_file.close()
         json_content = json.loads(data)
-        description = json_content['description']
+        description = json_content[constants.CAT_DESCRIPTION]
         assert description is not None
         os.remove(test_data_path + "test-443.json")
 
@@ -985,7 +932,7 @@ def test_issue_457(self):
         data = text_file.read()
         text_file.close()
         json_content = json.loads(data)
-        description = json_content['description']
+        description = json_content[constants.CAT_DESCRIPTION]
         assert description is not None
         os.remove(test_data_path + "test-457.json")
 
@@ -1007,5 +954,5 @@ def test_issue_445(self):
         data = text_file.read()
         json_content = json.loads(data)
         text_file.close()
-        assert len(json_content["acknowledgement"]) == 2
+        assert len(json_content[constants.CAT_ACKNOWLEDGEMENT]) == 2
         os.remove(test_data_path + "repositories/repos_oeg/test-445.json")
diff --git a/src/somef/test/test_data/repositories/wav2letter/CMakeLists.txt b/src/somef/test/test_data/repositories/wav2letter/CMakeLists.txt
new file mode 100644
index 00000000..69e4fb15
--- /dev/null
+++ b/src/somef/test/test_data/repositories/wav2letter/CMakeLists.txt
@@ -0,0 +1,168 @@
+cmake_minimum_required(VERSION 3.5.1)
+
+project(wav2letter++)
+
+# ----------------------------- Setup -----------------------------
+set(CMAKE_MODULE_PATH "${CMAKE_SOURCE_DIR}/cmake/")
+set(CMAKE_CXX_STANDARD 11)
+set(CMAKE_CXX_STANDARD_REQUIRED ON)
+set(CMAKE_CXX_EXTENSIONS OFF) # no compiler extensions like gnu++11
+
+# All include directories are based on namespace structure and use ./src
+# as their base directory
+include_directories(${CMAKE_SOURCE_DIR}/src)
+
+# ----------------------------- Configuration -----------------------------
+
+# Build tests
+option(BUILD_TESTS "Build tests for wav2letter++" ON)
+if (BUILD_TESTS)
+  enable_testing()
+  add_subdirectory(${CMAKE_SOURCE_DIR}/src/tests)
+endif ()
+
+# Build examples
+option(BUILD_EXAMPLES "Build examples for wav2letter++" ON)
+
+# ------------------------ Global External Dependencies ------------------------
+# ArrayFire
+find_package(ArrayFire 3.6.1 REQUIRED)
+if (ArrayFire_FOUND)
+  message(STATUS "ArrayFire found (include: ${ArrayFire_INCLUDE_DIRS}, library: ${ArrayFire_LIBRARIES})")
+  if (AFML_USE_UNIFIED)
+    # Set the AF_PATH environment variable to wherever the ArrayFire libs are
+    # located so they can be loaded in the unified backend
+    set(ENV{AF_PATH} ${ArrayFire_LIBRARIES})
+  endif ()
+else()
+  message(FATAL_ERROR "ArrayFire not found")
+endif()
+
+# Find GLog
+find_package(GLOG REQUIRED)
+if (GLOG_FOUND)
+  message(STATUS "GLOG found")
+else()
+  message(FATAL_ERROR "GLOG not found")
+endif()
+
+# Find GFlags
+find_package(GFLAGS REQUIRED)
+if (GFLAGS_FOUND)
+  message(STATUS "GFLAGS found")
+else()
+  message(FATAL_ERROR "GFLAGS not found")
+endif()
+
+# Find and setup OpenMP
+find_package(OpenMP)
+if (OPENMP_FOUND)
+  message(STATUS "OpenMP found")
+  set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} ${OpenMP_C_FLAGS}")
+  set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${OpenMP_CXX_FLAGS}")
+  set(CMAKE_EXE_LINKER_FLAGS
+    "${CMAKE_EXE_LINKER_FLAGS} ${OpenMP_EXE_LINKER_FLAGS}"
+  )
+else()
+  message(STATUS "OpenMP not found - building without OpenMP")
+endif()
+
+# ------------------------ flashlight ------------------------
+find_package(flashlight REQUIRED)
+if (flashlight_FOUND)
+  message("flashlight found (include: ${FLASHLIGHT_INCLUDE_DIRS} lib: flashlight::flashlight )")
+  if (NOT TARGET flashlight::Distributed)
+    message(FATAL_ERROR "flashlight must be build in distributed mode or wav2letter++")
+  else ()
+    message(STATUS "flashlight built in distributed mode.")
+  endif ()
+else()
+  message(FATAL_ERROR "flashlight not found.")
+endif ()
+
+# ------------------------ Components ------------------------
+
+# Common
+add_subdirectory(${CMAKE_CURRENT_SOURCE_DIR}/src/common)
+
+# Feature
+add_subdirectory(${CMAKE_CURRENT_SOURCE_DIR}/src/feature)
+
+# Runtime
+add_subdirectory(${CMAKE_CURRENT_SOURCE_DIR}/src/runtime)
+
+# Module
+add_subdirectory(${CMAKE_CURRENT_SOURCE_DIR}/src/module)
+
+# Decoder
+add_subdirectory(${CMAKE_CURRENT_SOURCE_DIR}/src/decoder)
+
+# Criterion
+add_subdirectory(${CMAKE_CURRENT_SOURCE_DIR}/src/criterion)
+
+# Data
+add_subdirectory(${CMAKE_CURRENT_SOURCE_DIR}/src/data)
+
+# ----------------------------- wav2letter++ lib  -----------------------------
+
+add_library(
+  wav2letter++
+  ""
+  )
+
+set_target_properties(
+  wav2letter++
+  PROPERTIES
+  LINKER_LANGUAGE CXX
+  CXX_STANDARD 11
+  )
+
+target_link_libraries(
+  wav2letter++
+  PUBLIC
+  common
+  criterion
+  data
+  decoder
+  module
+  runtime
+  )
+
+target_include_directories(
+  wav2letter++
+  PUBLIC
+  ${FLASHLIGHT_INCLUDE_DIRS} # includes are <flashlight/...>
+  )
+
+# ----------------------------- Train -----------------------------
+add_executable(
+  Train
+  Train.cpp
+)
+
+target_link_libraries(
+  Train
+  wav2letter++
+  )
+
+# ----------------------------- Test -----------------------------
+add_executable(
+  Test
+  Test.cpp
+)
+
+target_link_libraries(
+  Test
+  wav2letter++
+  )
+
+# ----------------------------- Decoder -----------------------------
+add_executable(
+  Decoder
+  Decode.cpp
+)
+
+target_link_libraries(
+  Decoder
+  wav2letter++
+  )
diff --git a/src/somef/test/test_data/repositories/wav2letter/CODE_OF_CONDUCT.md b/src/somef/test/test_data/repositories/wav2letter/CODE_OF_CONDUCT.md
new file mode 100644
index 00000000..0f7ad8bf
--- /dev/null
+++ b/src/somef/test/test_data/repositories/wav2letter/CODE_OF_CONDUCT.md
@@ -0,0 +1,5 @@
+# Code of Conduct
+
+Facebook has adopted a Code of Conduct that we expect project participants to adhere to.
+Please read the [full text](https://code.fb.com/codeofconduct/)
+so that you can understand what actions will and will not be tolerated.
diff --git a/src/somef/test/test_data/repositories/wav2letter/CONTRIBUTING.md b/src/somef/test/test_data/repositories/wav2letter/CONTRIBUTING.md
new file mode 100644
index 00000000..a9086c46
--- /dev/null
+++ b/src/somef/test/test_data/repositories/wav2letter/CONTRIBUTING.md
@@ -0,0 +1,27 @@
+# Contributing to wav2letter++
+We want to make contributing to this project as easy and transparent as
+possible.
+
+## Pull Requests
+We actively welcome your pull requests.
+
+1. Fork the repo and create your branch from `master`.
+2. If you've added code that should be tested, add tests.
+3. If you've changed APIs, update the documentation.
+4. Ensure the test suite passes.
+5. Make sure your code lints.
+6. If you haven't already, complete the Contributor License Agreement ("CLA").
+
+## Contributor License Agreement ("CLA")
+In order to accept your pull request, we need you to submit a CLA. You only need
+to do this once to work on any of Facebook's open source projects.
+
+Complete your CLA here: <https://code.facebook.com/cla>
+
+## Issues
+We use GitHub issues to track public bugs. Please ensure your description is
+clear and has sufficient instructions to be able to reproduce the issue.
+
+## License
+By contributing to wav2letter++, you agree that your contributions will be licensed
+under the LICENSE file in the root directory of this source tree.
diff --git a/src/somef/test/test_data/repositories/wav2letter/Decode.cpp b/src/somef/test/test_data/repositories/wav2letter/Decode.cpp
new file mode 100644
index 00000000..ae25bad3
--- /dev/null
+++ b/src/somef/test/test_data/repositories/wav2letter/Decode.cpp
@@ -0,0 +1,440 @@
+/**
+ * Copyright (c) Facebook, Inc. and its affiliates.
+ * All rights reserved.
+ *
+ * This source code is licensed under the BSD-style license found in the
+ * LICENSE file in the root directory of this source tree.
+ */
+
+#include <stdlib.h>
+#include <cstring>
+#include <fstream>
+#include <iomanip>
+#include <mutex>
+#include <string>
+#include <vector>
+
+#include <flashlight/flashlight.h>
+#include <gflags/gflags.h>
+#include <glog/logging.h>
+
+#include "common/Defines.h"
+#include "common/Dictionary.h"
+#include "common/Transforms.h"
+#include "common/Utils.h"
+#include "criterion/criterion.h"
+#include "data/Featurize.h"
+#include "data/W2lDataset.h"
+#include "data/W2lNumberedFilesDataset.h"
+#include "decoder/Decoder.hpp"
+#include "decoder/KenLM.hpp"
+#include "decoder/Trie.hpp"
+#include "module/module.h"
+#include "runtime/Logger.h"
+#include "runtime/Serial.h"
+
+#ifdef BUILD_FB_DEPENDENCIES
+#include "fb/W2lEverstoreDataset.h"
+#endif
+
+using namespace w2l;
+
+int main(int argc, char** argv) {
+  google::InitGoogleLogging(argv[0]);
+  std::string exec(argv[0]);
+  std::vector<std::string> argvs;
+  for (int i = 0; i < argc; i++) {
+    argvs.emplace_back(argv[i]);
+  }
+  gflags::SetUsageMessage(
+      "Usage: \n " + exec + " [data_path] [dataset_name] [flags]");
+  if (argc <= 1) {
+    LOG(FATAL) << gflags::ProgramUsage();
+  }
+
+  /* ===================== Parse Options ===================== */
+  LOG(INFO) << "Parsing command line flags";
+  gflags::ParseCommandLineFlags(&argc, &argv, false);
+  if (!FLAGS_flagsfile.empty()) {
+    LOG(INFO) << "Reading flags from file " << FLAGS_flagsfile;
+    gflags::ReadFromFlagsFile(FLAGS_flagsfile, argv[0], true);
+  }
+
+  /* ===================== Create Network ===================== */
+  std::shared_ptr<fl::Module> network;
+  std::shared_ptr<fl::SequenceCriterion> criterion;
+  if (!FLAGS_am.empty()) {
+    std::unordered_map<std::string, std::string> cfg;
+    LOG(INFO) << "[Network] Reading acoustic model from " << FLAGS_am;
+
+    W2lSerializer::load(FLAGS_am, cfg, network, criterion);
+    network->eval();
+
+    LOG(INFO) << "[Network] " << network->prettyString();
+    LOG(INFO) << "[Network] Number of params: " << numTotalParams(network);
+
+    auto flags = cfg.find(kGflags);
+    if (flags == cfg.end()) {
+      LOG(FATAL) << "[Network] Invalid config loaded from " << FLAGS_am;
+    }
+    LOG(INFO) << "[Network] Updating flags from config file: " << FLAGS_am;
+    gflags::ReadFlagsFromString(flags->second, gflags::GetArgv0(), true);
+    gflags::ParseCommandLineFlags(&argc, &argv, false);
+  }
+
+  LOG(INFO) << "Gflags after parsing \n" << serializeGflags("; ");
+
+  /* ===================== Create Dictionary ===================== */
+  auto word2spell = loadWords(FLAGS_lexicon, FLAGS_maxword);
+
+  auto letterDict = makeDictionary(FLAGS_tokens);
+  int numClasses = letterDict.indexSize();
+  LOG(INFO) << "Number of classes (network): " << numClasses;
+
+  Dictionary wordDict;
+  for (auto& it : word2spell) {
+    wordDict.addToken(it.first);
+  }
+  wordDict.setDefaultIndex(wordDict.getIndex(kUnkToken));
+  LOG(INFO) << "Number of words: " << wordDict.indexSize();
+
+  DictionaryMap dicts;
+  dicts.insert({kTargetIdx, letterDict});
+  dicts.insert({kWordIdx, wordDict});
+
+  /* ===================== Create Dataset ===================== */
+  if (!(FLAGS_am.empty() ^ FLAGS_emission_dir.empty())) {
+    LOG(FATAL)
+        << "One and only one of flag -am and -emission_dir should be set.";
+  }
+
+  auto emissionSet = std::make_shared<EmissionSet>();
+
+  /* Using acoustic model */
+  if (!FLAGS_am.empty()) {
+    // Load dataset
+    int worldRank = 0;
+    int worldSize = 1;
+    std::unique_ptr<W2lDataset> ds;
+    if (FLAGS_everstoredb) {
+#ifdef BUILD_FB_DEPENDENCIES
+      W2lEverstoreDataset::init(); // Required for everstore client
+      ds = std::unique_ptr<W2lEverstoreDataset>(new W2lEverstoreDataset(
+          FLAGS_test, dicts, 1, worldRank, worldSize, FLAGS_targettype));
+#else
+      LOG(FATAL) << "W2lEverstoreDataset not supported: "
+                 << "build with -DBUILD_FB_DEPENDENCIES";
+#endif
+    } else {
+      ds = std::unique_ptr<W2lNumberedFilesDataset>(new W2lNumberedFilesDataset(
+          FLAGS_test, dicts, 1, worldRank, worldSize, FLAGS_datadir));
+    }
+    ds->shuffle(3);
+    LOG(INFO) << "[Serialization] Running forward pass ...";
+
+    int cnt = 0;
+    for (auto& sample : *ds) {
+      auto rawEmission = network->forward(fl::input(sample[kInputIdx]));
+      int N = rawEmission.dims(0);
+      int T = rawEmission.dims(1);
+
+      auto emission = afToVector<float>(rawEmission);
+      auto ltrTarget = afToVector<int>(sample[kTargetIdx]);
+      auto wrdTarget = afToVector<int>(sample[kWordIdx]);
+      emissionSet->emissions.emplace_back(emission);
+      emissionSet->wordTargets.emplace_back(wrdTarget);
+      emissionSet->letterTargets.emplace_back(ltrTarget);
+      emissionSet->emissionT.emplace_back(T);
+      emissionSet->emissionN = N;
+      if (FLAGS_criterion == kAsgCriterion) {
+        emissionSet->transition =
+            afToVector<float>(criterion->param(0).array());
+      }
+
+      ++cnt;
+      if (cnt == FLAGS_maxload) {
+        break;
+      }
+    }
+  }
+  /* Using existing emissions */
+  else {
+    std::string loadPath = pathsConcat(FLAGS_emission_dir, FLAGS_test + ".bin");
+    LOG(INFO) << "[Serialization] Loading file: " << loadPath;
+    W2lSerializer::load(loadPath, emissionSet);
+  }
+
+  int nSample = emissionSet->emissions.size();
+  nSample = FLAGS_maxload > 0 ? std::min(nSample, FLAGS_maxload) : nSample;
+  int nSamplePerThread =
+      std::ceil(nSample / static_cast<float>(FLAGS_nthread_decoder));
+  LOG(INFO) << "[Dataset] Number of samples per thread: " << nSamplePerThread;
+
+  /* ===================== Decode ===================== */
+  // Prepare counters
+  std::vector<double> sliceWer(FLAGS_nthread_decoder);
+  std::vector<double> sliceLer(FLAGS_nthread_decoder);
+  std::vector<int> sliceNumWords(FLAGS_nthread_decoder, 0);
+  std::vector<int> sliceNumLetters(FLAGS_nthread_decoder, 0);
+  std::vector<int> sliceNumSamples(FLAGS_nthread_decoder, 0);
+  std::vector<double> sliceTime(FLAGS_nthread_decoder, 0);
+
+  // Prepare criterion
+  ModelType modelType = ModelType::ASG;
+  if (FLAGS_criterion == kCtcCriterion) {
+    modelType = ModelType::CTC;
+  } else if (FLAGS_criterion != kAsgCriterion) {
+    LOG(FATAL) << "[Decoder] Invalid model type: " << FLAGS_criterion;
+  }
+
+  const auto& transition = emissionSet->transition;
+
+  // Prepare decoder options
+  DecoderOptions decoderOpt(
+      FLAGS_beamsize,
+      static_cast<float>(FLAGS_beamscore),
+      static_cast<float>(FLAGS_lmweight),
+      static_cast<float>(FLAGS_wordscore),
+      static_cast<float>(FLAGS_unkweight),
+      FLAGS_forceendsil,
+      FLAGS_logadd,
+      static_cast<float>(FLAGS_silweight),
+      modelType);
+
+  // Prepare log writer
+  std::mutex hypMutex, refMutex, logMutex;
+  std::ofstream hypStream, refStream, logStream;
+  if (!FLAGS_sclite.empty()) {
+    auto fileName = cleanFilepath(FLAGS_test);
+    auto hypPath = pathsConcat(FLAGS_sclite, fileName + ".hyp");
+    auto refPath = pathsConcat(FLAGS_sclite, fileName + ".ref");
+    auto logPath = pathsConcat(FLAGS_sclite, fileName + ".log");
+    hypStream.open(hypPath);
+    refStream.open(refPath);
+    logStream.open(logPath);
+    if (!hypStream.is_open() || !hypStream.good()) {
+      LOG(FATAL) << "Error opening hypothesis file: " << hypPath;
+    }
+    if (!refStream.is_open() || !refStream.good()) {
+      LOG(FATAL) << "Error opening reference file: " << refPath;
+    }
+    if (!logStream.is_open() || !logStream.good()) {
+      LOG(FATAL) << "Error opening log file: " << logPath;
+    }
+  }
+
+  auto writeHyp = [&](const std::string& hypStr) {
+    std::lock_guard<std::mutex> lock(hypMutex);
+    hypStream << hypStr;
+  };
+  auto writeRef = [&](const std::string& refStr) {
+    std::lock_guard<std::mutex> lock(refMutex);
+    refStream << refStr;
+  };
+  auto writeLog = [&](const std::string& logStr) {
+    std::lock_guard<std::mutex> lock(logMutex);
+    logStream << logStr;
+  };
+
+  // Decoding
+  auto runDecoder = [&](int tid, int start, int end) {
+    try {
+      // Build Language Model
+      std::shared_ptr<LM> lm;
+      if (FLAGS_lmtype == "kenlm") {
+        lm = std::make_shared<KenLM>(FLAGS_lm);
+        if (!lm) {
+          LOG(FATAL) << "[LM constructing] Failed to load LM: " << FLAGS_lm;
+        }
+      } else {
+        LOG(FATAL) << "[LM constructing] Invalid LM Type: " << FLAGS_lmtype;
+      }
+      LOG(INFO) << "[Decoder] LM constructed.\n";
+
+      // Build Trie
+      if (std::strlen(kSilToken) != 1) {
+        LOG(FATAL) << "[Decoder] Invalid unknown_symbol: " << kSilToken;
+      }
+      if (std::strlen(kBlankToken) != 1) {
+        LOG(FATAL) << "[Decoder] Invalid unknown_symbol: " << kBlankToken;
+      }
+      int silIdx = letterDict.getIndex(kSilToken);
+      int blankIdx = FLAGS_criterion == kCtcCriterion
+          ? letterDict.getIndex(kBlankToken)
+          : -1;
+      int unkIdx = lm->index(kUnkToken);
+      std::shared_ptr<Trie> trie =
+          std::make_shared<Trie>(letterDict.indexSize(), silIdx);
+      auto start_state = lm->start(false);
+
+      for (auto& it : word2spell) {
+        std::string word = it.first;
+        int lmIdx = lm->index(word);
+        if (lmIdx == unkIdx) { // We don't insert unknown words
+          continue;
+        }
+        float score;
+        auto dummyState = lm->score(start_state, lmIdx, score);
+        for (auto& spelling : it.second) {
+          auto spellingTensor = spelling2tensor(spelling, letterDict);
+          trie->insert(
+              spellingTensor,
+              std::make_shared<TrieLabel>(lmIdx, wordDict.getIndex(word)),
+              score);
+        }
+      }
+      LOG(INFO) << "[Decoder] Trie planted.\n";
+
+      // Smearing
+      SmearingMode smear_mode = SmearingMode::NONE;
+      if (FLAGS_smearing == "logadd") {
+        smear_mode = SmearingMode::LOGADD;
+      } else if (FLAGS_smearing == "max") {
+        smear_mode = SmearingMode::MAX;
+      } else if (FLAGS_smearing != "none") {
+        LOG(FATAL) << "[Decoder] Invalid smearing mode: " << FLAGS_smearing;
+      }
+      trie->smear(smear_mode);
+      LOG(INFO) << "[Decoder] Trie smeared.\n";
+
+      // Build Decoder
+      std::shared_ptr<TrieLabel> unk =
+          std::make_shared<TrieLabel>(unkIdx, wordDict.getIndex(kUnkToken));
+      Decoder decoder(trie, lm, silIdx, blankIdx, unk);
+      LOG(INFO) << "[Decoder] Decoder loaded in thread: " << tid;
+
+      // Get data and run decoder
+      TestMeters meters;
+      int sliceSize = end - start;
+      meters.timer.resume();
+      for (int s = start; s < end; s++) {
+        auto emission = emissionSet->emissions[s];
+        auto wordTarget = emissionSet->wordTargets[s];
+        auto letterTarget = emissionSet->letterTargets[s];
+        auto T = emissionSet->emissionT[s];
+        auto N = emissionSet->emissionN;
+
+        std::vector<float> score;
+        std::vector<std::vector<int>> wordPredictions;
+        std::vector<std::vector<int>> letterPredictions;
+
+        std::tie(score, wordPredictions, letterPredictions) = decoder.decode(
+            decoderOpt, transition.data(), emission.data(), T, N);
+
+        // Cleanup predictions
+        auto wordPrediction = wordPredictions[0];
+        auto letterPrediction = letterPredictions[0];
+        if (FLAGS_criterion == kCtcCriterion ||
+            FLAGS_criterion == kAsgCriterion) {
+          uniq(letterPrediction);
+        }
+        if (FLAGS_criterion == kCtcCriterion) {
+          letterPrediction.erase(
+              std::remove(
+                  letterPrediction.begin(), letterPrediction.end(), blankIdx),
+              letterPrediction.end());
+        }
+        remapLabels(letterTarget, letterDict);
+        remapLabels(letterPrediction, letterDict);
+        validateWords(wordPrediction, wordDict.getIndex(kUnkToken));
+
+        // Update meters & print out predictions
+        meters.werSlice.add(wordPrediction, wordTarget);
+        meters.lerSlice.add(letterPrediction, letterTarget);
+
+        if (FLAGS_show) {
+          meters.wer.reset();
+          meters.ler.reset();
+          meters.wer.add(wordPrediction, wordTarget);
+          meters.ler.add(letterPrediction, letterTarget);
+
+          auto wordTargetStr = tensor2words(wordTarget, wordDict);
+          auto wordPredictionStr = tensor2words(wordPrediction, wordDict);
+
+          std::stringstream buffer;
+          buffer << "|T|: " << wordTargetStr << std::endl;
+          buffer << "|P|: " << wordPredictionStr << std::endl;
+          if (FLAGS_showletters) {
+            buffer << "|t|: " << tensor2letters(letterTarget, letterDict)
+                   << std::endl;
+            buffer << "|p|: " << tensor2letters(letterPrediction, letterDict)
+                   << std::endl;
+          }
+          buffer << "[sample: " << s << ", WER: " << meters.wer.value()[0]
+                 << "\%, LER: " << meters.ler.value()[0]
+                 << "\%, slice WER: " << meters.werSlice.value()[0]
+                 << "\%, slice LER: " << meters.lerSlice.value()[0]
+                 << "\%, progress: "
+                 << static_cast<float>(s - start + 1) / sliceSize * 100 << "\%]"
+                 << std::endl;
+
+          std::cout << buffer.str();
+          if (!FLAGS_sclite.empty()) {
+            std::string suffix = "(SPEAKER_" + std::to_string(s) + ")\n";
+            writeHyp(wordPredictionStr + suffix);
+            writeRef(wordTargetStr + suffix);
+            writeLog(buffer.str());
+          }
+        }
+
+        // Update conters
+        sliceNumWords[tid] += wordTarget.size();
+        sliceNumLetters[tid] += letterTarget.size();
+      }
+      meters.timer.stop();
+      sliceWer[tid] = meters.werSlice.value()[0];
+      sliceLer[tid] = meters.lerSlice.value()[0];
+      sliceNumSamples[tid] = sliceSize;
+      sliceTime[tid] = meters.timer.value();
+    } catch (const std::exception& exc) {
+      LOG(FATAL) << "Exception in thread " << tid << "\n" << exc.what();
+    }
+  };
+
+  /* Spread threades */
+  auto startThreads = [&]() {
+    fl::ThreadPool threadPool(FLAGS_nthread_decoder);
+    for (int i = 0; i < FLAGS_nthread_decoder; i++) {
+      int start = i * nSamplePerThread;
+      if (start >= nSample) {
+        break;
+      }
+      int end = std::min((i + 1) * nSamplePerThread, nSample);
+      threadPool.enqueue(runDecoder, i, start, end);
+    }
+  };
+  auto timer = fl::TimeMeter();
+  timer.resume();
+  startThreads();
+  timer.stop();
+
+  /* Compute statistics */
+  int totalLetters = 0, totalWords = 0, totalSamples = 0;
+  for (int i = 0; i < FLAGS_nthread_decoder; i++) {
+    totalLetters += sliceNumLetters[i];
+    totalWords += sliceNumWords[i];
+    totalSamples += sliceNumSamples[i];
+  }
+  double totalWer = 0, totalLer = 0, totalTime = 0;
+  for (int i = 0; i < FLAGS_nthread_decoder; i++) {
+    totalWer += sliceWer[i] * sliceNumWords[i] / totalWords;
+    totalLer += sliceLer[i] * sliceNumLetters[i] / totalLetters;
+    totalTime += sliceTime[i];
+  }
+
+  std::stringstream buffer;
+  buffer << "------\n";
+  buffer << "[Decode " << FLAGS_test << " (" << totalSamples << " samples) in "
+         << timer.value() << "s (actual decoding time " << std::setprecision(3)
+         << totalTime / totalSamples
+         << "s/sample) -- WER: " << std::setprecision(6) << totalWer
+         << ", LER: " << totalLer << "]" << std::endl;
+  LOG(INFO) << buffer.str();
+  if (!FLAGS_sclite.empty()) {
+    writeLog(buffer.str());
+    hypStream.close();
+    refStream.close();
+    logStream.close();
+  }
+  return 0;
+}
diff --git a/src/somef/test/test_data/repositories/wav2letter/LICENSE b/src/somef/test/test_data/repositories/wav2letter/LICENSE
new file mode 100644
index 00000000..d3eb5243
--- /dev/null
+++ b/src/somef/test/test_data/repositories/wav2letter/LICENSE
@@ -0,0 +1,30 @@
+BSD License
+
+For wav2letter++ software
+
+Copyright (c) Facebook, Inc. and its affiliates. All rights reserved.
+
+Redistribution and use in source and binary forms, with or without modification,
+are permitted provided that the following conditions are met:
+
+ * Redistributions of source code must retain the above copyright notice, this
+   list of conditions and the following disclaimer.
+
+ * Redistributions in binary form must reproduce the above copyright notice,
+   this list of conditions and the following disclaimer in the documentation
+   and/or other materials provided with the distribution.
+
+ * Neither the name Facebook nor the names of its contributors may be used to
+   endorse or promote products derived from this software without specific
+   prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
+ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR
+ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
+ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
diff --git a/src/somef/test/test_data/repositories/wav2letter/README.md b/src/somef/test/test_data/repositories/wav2letter/README.md
new file mode 100644
index 00000000..92d31f09
--- /dev/null
+++ b/src/somef/test/test_data/repositories/wav2letter/README.md
@@ -0,0 +1,39 @@
+# wav2letter++
+
+wav2letter++ is a fast open source speech processing toolkit from Facebook AI Research.
+It is written entirely in C++ and uses the [ArrayFire](https://github.com/arrayfire/arrayfire) tensor library and the [flashlight](https://github.com/facebookresearch/flashlight) machine learning library for maximum efficiency.
+Our approach is detailed in this [arXiv paper](https://arxiv.org/abs/1812.07625).
+
+## Building wav2letter++
+See [Building Instructions](docs/installation.md) for details.
+
+## Full documentation
+- [Training](docs/train.md)
+- [Testing / Decoding](docs/decoder.md)
+
+We also provide complete recipes for WSJ, Timit and Librispeech and they be found in `recipes` folder.
+
+# Citation
+
+If you use the code in your paper, then please cite it as:
+
+```
+@article{pratap2018w2l,
+  author          = {Vineel Pratap, Awni Hannun, Qiantong Xu, Jeff Cai, Jacob Kahn, Gabriel Synnaeve, Vitaliy Liptchinsky, Ronan Collobert},
+  title           = {wav2letter++: The Fastest Open-source Speech Recognition System},
+  journal         = {CoRR},
+  volume          = {abs/1812.07625},
+  year            = {2018},
+  url             = {https://arxiv.org/abs/1812.07625},
+}
+```
+
+## Join the wav2letter community
+* Facebook page: https://www.facebook.com/groups/717232008481207/
+* Google group: https://groups.google.com/forum/#!forum/wav2letter-users
+* Contact: vineelkpratap@fb.com, awni@fb.com, qiantong@fb.com, jcai@fb.com, jacobkahn@fb.com, gab@fb.com, vitaliy888@fb.com, locronan@fb.com
+
+See the [CONTRIBUTING](CONTRIBUTING.md) file for how to help out.
+
+## License
+wav2letter++ is BSD-licensed, as found in the [LICENSE](LICENSE) file.
diff --git a/src/somef/test/test_data/repositories/wav2letter/Test.cpp b/src/somef/test/test_data/repositories/wav2letter/Test.cpp
new file mode 100644
index 00000000..696ad003
--- /dev/null
+++ b/src/somef/test/test_data/repositories/wav2letter/Test.cpp
@@ -0,0 +1,184 @@
+/**
+ * Copyright (c) Facebook, Inc. and its affiliates.
+ * All rights reserved.
+ *
+ * This source code is licensed under the BSD-style license found in the
+ * LICENSE file in the root directory of this source tree.
+ */
+
+#include <stdlib.h>
+#include <fstream>
+#include <string>
+#include <vector>
+
+#include <flashlight/flashlight.h>
+#include <gflags/gflags.h>
+#include <glog/logging.h>
+
+#include "common/Defines.h"
+#include "common/Dictionary.h"
+#include "common/Transforms.h"
+#include "common/Utils.h"
+#include "criterion/criterion.h"
+#include "data/W2lDataset.h"
+#include "data/W2lNumberedFilesDataset.h"
+#include "module/module.h"
+#include "runtime/Logger.h"
+#include "runtime/Serial.h"
+
+using namespace w2l;
+
+int main(int argc, char** argv) {
+  google::InitGoogleLogging(argv[0]);
+  std::string exec(argv[0]);
+  std::vector<std::string> argvs;
+  for (int i = 0; i < argc; i++) {
+    argvs.emplace_back(argv[i]);
+  }
+  gflags::SetUsageMessage(
+      "Usage: \n " + exec + " [data_path] [dataset_name] [flags]");
+  if (argc <= 1) {
+    LOG(FATAL) << gflags::ProgramUsage();
+  }
+
+  /* ===================== Parse Options ===================== */
+  LOG(INFO) << "Parsing command line flags";
+  gflags::ParseCommandLineFlags(&argc, &argv, false);
+  if (!FLAGS_flagsfile.empty()) {
+    LOG(INFO) << "Reading flags from file " << FLAGS_flagsfile;
+    gflags::ReadFromFlagsFile(FLAGS_flagsfile, argv[0], true);
+  }
+
+  /* ===================== Create Network ===================== */
+  std::shared_ptr<fl::Module> network;
+  std::shared_ptr<fl::SequenceCriterion> criterion;
+  std::unordered_map<std::string, std::string> cfg;
+  LOG(INFO) << "[Network] Reading acoustic model from " << FLAGS_am;
+  W2lSerializer::load(FLAGS_am, cfg, network, criterion);
+  network->eval();
+  criterion->eval();
+
+  LOG(INFO) << "[Network] " << network->prettyString();
+  LOG(INFO) << "[Criterion] " << criterion->prettyString();
+  LOG(INFO) << "[Network] Number of params: " << numTotalParams(network);
+
+  auto flags = cfg.find(kGflags);
+  if (flags == cfg.end()) {
+    LOG(FATAL) << "[Network] Invalid config loaded from " << FLAGS_am;
+  }
+  LOG(INFO) << "[Network] Updating flags from config file: " << FLAGS_am;
+  gflags::ReadFlagsFromString(flags->second, gflags::GetArgv0(), true);
+  gflags::ParseCommandLineFlags(&argc, &argv, false);
+
+  LOG(INFO) << "Gflags after parsing \n" << serializeGflags("; ");
+
+  /* ===================== Create Dictionary ===================== */
+  auto word2spell = loadWords(FLAGS_lexicon, FLAGS_maxword);
+
+  auto letterDict = makeDictionary(FLAGS_tokens);
+  int numClasses = letterDict.indexSize();
+  LOG(INFO) << "Number of classes (network): " << numClasses;
+
+  Dictionary wordDict;
+  for (auto& it : word2spell) {
+    wordDict.addToken(it.first);
+  }
+  wordDict.setDefaultIndex(wordDict.getIndex(kUnkToken));
+  LOG(INFO) << "Number of words: " << wordDict.indexSize();
+
+  DictionaryMap dicts;
+  dicts.insert({kTargetIdx, letterDict});
+  dicts.insert({kWordIdx, wordDict});
+
+  /* ===================== Create Dataset ===================== */
+  int worldRank = 0;
+  int worldSize = 1;
+  W2lNumberedFilesDataset ds(
+      FLAGS_test, dicts, 1, worldRank, worldSize, FLAGS_datadir);
+  ds.shuffle(3);
+  int nSamples = ds.size();
+  if (FLAGS_maxload > 0) {
+    nSamples = std::min(nSamples, FLAGS_maxload);
+  }
+  LOG(INFO) << "[Dataset] Dataset loaded.";
+
+  /* ===================== Test ===================== */
+  TestMeters meters;
+
+  auto emissionSet = std::make_shared<EmissionSet>();
+  meters.timer.resume();
+  int cnt = 1;
+  for (auto& sample : ds) {
+    auto rawEmission = network->forward(fl::input(sample[kInputIdx]));
+    auto emission = afToVector<float>(rawEmission);
+    auto ltrTarget = afToVector<int>(sample[kTargetIdx]);
+    auto wrdTarget = afToVector<int>(sample[kWordIdx]);
+
+    /* viterbiPath + remove duplication/blank */
+    auto viterbiPath =
+        afToVector<int>(criterion->viterbiPath(rawEmission.array()));
+    if (FLAGS_criterion == kCtcCriterion || FLAGS_criterion == kAsgCriterion) {
+      uniq(viterbiPath);
+    }
+    if (FLAGS_criterion == kCtcCriterion) {
+      auto blankidx = letterDict.getIndex(kBlankToken);
+      viterbiPath.erase(
+          std::remove(viterbiPath.begin(), viterbiPath.end(), blankidx),
+          viterbiPath.end());
+    }
+    remapLabels(viterbiPath, letterDict);
+    remapLabels(ltrTarget, letterDict);
+
+    meters.lerSlice.add(viterbiPath, ltrTarget);
+
+    auto wordViterbi = ltrTensor2wrdTensor(
+        viterbiPath, wordDict, letterDict, letterDict.getIndex(kSilToken));
+
+    meters.werSlice.add(wordViterbi, wrdTarget);
+
+    if (FLAGS_show) {
+      meters.ler.reset();
+      meters.wer.reset();
+      meters.ler.add(viterbiPath, ltrTarget);
+      meters.wer.add(wordViterbi, wrdTarget);
+
+      std::cout << "|T|: " << tensor2letters(ltrTarget, letterDict)
+                << std::endl;
+      std::cout << "|P|: " << tensor2letters(viterbiPath, letterDict)
+                << std::endl;
+      std::cout << "[sample: " << cnt << ", WER: " << meters.wer.value()[0]
+                << "\%, LER: " << meters.ler.value()[0]
+                << "\%, total WER: " << meters.werSlice.value()[0]
+                << "\%, total LER: " << meters.lerSlice.value()[0]
+                << "\%, progress: " << static_cast<float>(cnt) / nSamples * 100
+                << "\%]" << std::endl;
+      ++cnt;
+      if (cnt == FLAGS_maxload) {
+        break;
+      }
+    }
+
+    /* Save emission and targets */
+    int N = rawEmission.dims(0);
+    int T = rawEmission.dims(1);
+    emissionSet->emissions.emplace_back(emission);
+    emissionSet->letterTargets.emplace_back(ltrTarget);
+    emissionSet->wordTargets.emplace_back(wrdTarget);
+    emissionSet->emissionT.emplace_back(T);
+    emissionSet->emissionN = N;
+  }
+  if (FLAGS_criterion == kAsgCriterion) {
+    emissionSet->transition = afToVector<float>(criterion->param(0).array());
+  }
+  meters.timer.stop();
+  std::cout << "---\n[total WER: " << meters.werSlice.value()[0]
+            << "\%, total LER: " << meters.lerSlice.value()[0]
+            << "\%, time: " << meters.timer.value() << "s]" << std::endl;
+
+  /* ====== Serialize emission and targets for decoding ====== */
+  std::string savePath = pathsConcat(FLAGS_emission_dir, FLAGS_test + ".bin");
+  LOG(INFO) << "[Serialization] Saving into file: " << savePath;
+  W2lSerializer::save(savePath, emissionSet);
+
+  return 0;
+}
diff --git a/src/somef/test/test_data/repositories/wav2letter/Train.cpp b/src/somef/test/test_data/repositories/wav2letter/Train.cpp
new file mode 100644
index 00000000..22b912c1
--- /dev/null
+++ b/src/somef/test/test_data/repositories/wav2letter/Train.cpp
@@ -0,0 +1,564 @@
+/**
+ * Copyright (c) Facebook, Inc. and its affiliates.
+ * All rights reserved.
+ *
+ * This source code is licensed under the BSD-style license found in the
+ * LICENSE file in the root directory of this source tree.
+ */
+
+#include <stdlib.h>
+#include <fstream>
+#include <regex>
+#include <string>
+#include <vector>
+
+#include <cereal/archives/json.hpp>
+#include <cereal/types/unordered_map.hpp>
+#include <flashlight/flashlight.h>
+#include <gflags/gflags.h>
+#include <glog/logging.h>
+
+#include "common/Defines.h"
+#include "common/Dictionary.h"
+#include "common/Transforms.h"
+#include "common/Utils.h"
+#include "criterion/criterion.h"
+#include "data/Featurize.h"
+#include "data/W2lDataset.h"
+#include "data/W2lNumberedFilesDataset.h"
+#include "module/module.h"
+#include "runtime/runtime.h"
+
+#ifdef BUILD_FB_DEPENDENCIES
+#include "fb/W2lEverstoreDataset.h"
+#endif
+
+using namespace w2l;
+
+int main(int argc, char** argv) {
+  google::InitGoogleLogging(argv[0]);
+  std::string exec(argv[0]);
+  std::vector<std::string> argvs;
+  for (int i = 0; i < argc; i++) {
+    argvs.emplace_back(argv[i]);
+  }
+  gflags::SetUsageMessage(
+      "Usage: \n " + exec + " train [flags]\n or " + std::string() +
+      " continue [directory] [flags]\n or " + std::string(argv[0]) +
+      " fork [directory/model] [flags]");
+
+  /* ===================== Parse Options ===================== */
+  int runidx = 1; // current #runs in this path
+  std::string runpath; // current experiment path
+  std::string reloadpath; // path to model to reload
+  int startEpoch = 0;
+  if (argc <= 1) {
+    LOG(FATAL) << gflags::ProgramUsage();
+  }
+  if (strcmp(argv[1], "train") == 0) {
+    LOG(INFO) << "Parsing command line flags";
+    gflags::ParseCommandLineFlags(&argc, &argv, false);
+    if (!FLAGS_flagsfile.empty()) {
+      LOG(INFO) << "Reading flags from file " << FLAGS_flagsfile;
+      gflags::ReadFromFlagsFile(FLAGS_flagsfile, argv[0], true);
+    }
+    runpath = newRunPath(FLAGS_rundir, FLAGS_runname, FLAGS_tag);
+  } else if (strcmp(argv[1], "continue") == 0) {
+    runpath = argv[2];
+    while (fileExists(getRunFile("model_last.bin", runidx, runpath))) {
+      ++runidx;
+    }
+    reloadpath = getRunFile("model_last.bin", runidx - 1, runpath);
+    LOG(INFO) << "reload path is " << reloadpath;
+    std::unordered_map<std::string, std::string> cfg;
+    W2lSerializer::load(reloadpath, cfg);
+    auto flags = cfg.find(kGflags);
+    if (flags == cfg.end()) {
+      LOG(FATAL) << "Invalid config loaded from " << reloadpath;
+    }
+    LOG(INFO) << "Reading flags from config file " << reloadpath;
+    gflags::ReadFlagsFromString(flags->second, gflags::GetArgv0(), true);
+    if (argc > 3) {
+      LOG(INFO) << "Parsing command line flags";
+      LOG(INFO) << "Overriding flags should be mutable when using `continue`";
+      gflags::ParseCommandLineFlags(&argc, &argv, false);
+    }
+    if (!FLAGS_flagsfile.empty()) {
+      LOG(INFO) << "Reading flags from file " << FLAGS_flagsfile;
+      gflags::ReadFromFlagsFile(FLAGS_flagsfile, argv[0], true);
+    }
+    auto epoch = cfg.find(kEpoch);
+    if (epoch == cfg.end()) {
+      LOG(WARNING) << "Did not find epoch to start from, starting from 0.";
+    } else {
+      startEpoch = std::stoi(epoch->second);
+    }
+  } else if (strcmp(argv[1], "fork") == 0) {
+    reloadpath = argv[2];
+    std::unordered_map<std::string, std::string> cfg;
+    W2lSerializer::load(reloadpath, cfg);
+    auto flags = cfg.find(kGflags);
+    if (flags == cfg.end()) {
+      LOG(FATAL) << "Invalid config loaded from " << reloadpath;
+    }
+
+    LOG(INFO) << "Reading flags from config file " << reloadpath;
+    gflags::ReadFlagsFromString(flags->second, gflags::GetArgv0(), true);
+
+    if (argc > 3) {
+      LOG(INFO) << "Parsing command line flags";
+      LOG(INFO) << "Overriding flags should be mutable when using `fork`";
+      gflags::ParseCommandLineFlags(&argc, &argv, false);
+    }
+
+    if (!FLAGS_flagsfile.empty()) {
+      LOG(INFO) << "Reading flags from file" << FLAGS_flagsfile;
+      gflags::ReadFromFlagsFile(FLAGS_flagsfile, argv[0], true);
+    }
+    runpath = newRunPath(FLAGS_rundir, FLAGS_runname, FLAGS_tag);
+  } else {
+    LOG(FATAL) << gflags::ProgramUsage();
+  }
+
+  af::setMemStepSize(FLAGS_memstepsize);
+  af::setSeed(FLAGS_seed);
+  af::setFFTPlanCacheSize(FLAGS_fftcachesize);
+
+  maybeInitDistributedEnv(
+      FLAGS_enable_distributed,
+      FLAGS_world_rank,
+      FLAGS_world_size,
+      FLAGS_rndv_filepath);
+
+  auto worldRank = fl::getWorldRank();
+  auto worldSize = fl::getWorldSize();
+
+  bool isMaster = (worldRank == 0);
+
+  LOG_MASTER(INFO) << "Gflags after parsing \n" << serializeGflags("; ");
+
+  LOG_MASTER(INFO) << "Experiment path: " << runpath;
+  LOG_MASTER(INFO) << "Experiment runidx: " << runidx;
+
+  std::unordered_map<std::string, std::string> config = {
+      {kProgramName, exec},
+      {kCommandLine, join(" ", argvs)},
+      {kGflags, serializeGflags()},
+      // extra goodies
+      {kUserName, getEnvVar("USER")},
+      {kHostName, getEnvVar("HOSTNAME")},
+      {kTimestamp, getCurrentDate() + ", " + getCurrentDate()},
+      {kRunIdx, std::to_string(runidx)},
+      {kRunPath, runpath}};
+
+  auto validsets = split(',', trim(FLAGS_valid));
+
+  /* ===================== Create Dictionary ===================== */
+  Dictionary dict = makeDictionary();
+  int numClasses = dict.indexSize();
+  LOG_MASTER(INFO) << "Number of classes (network) = " << numClasses;
+
+  DictionaryMap dicts;
+  dicts.insert({kTargetIdx, dict});
+
+  /* ===================== Create Network ===================== */
+  std::shared_ptr<fl::Module> network;
+  std::shared_ptr<fl::SequenceCriterion> criterion;
+  auto scalemode = getCriterionScaleMode(FLAGS_onorm, FLAGS_sqnorm);
+  if (reloadpath.empty()) {
+    auto archfile = pathsConcat(FLAGS_archdir, FLAGS_arch);
+    LOG_MASTER(INFO) << "Loading architecture file from " << archfile;
+    auto numFeatures = getSpeechFeatureSize();
+    // Encoder network, works on audio
+    network = createW2lSeqModule(archfile, numFeatures, numClasses);
+
+    if (FLAGS_criterion == kCtcCriterion) {
+      criterion = std::make_shared<fl::CTCLoss>(scalemode);
+    } else if (FLAGS_criterion == kAsgCriterion) {
+      criterion =
+          std::make_shared<fl::ASGLoss>(numClasses, scalemode, FLAGS_transdiag);
+    } else if (FLAGS_criterion == kSeq2SeqCriterion) {
+      criterion = std::make_shared<fl::Seq2SeqCriterion>(
+          buildSeq2Seq(numClasses, dict.getIndex(kEosToken)));
+    } else {
+      LOG(FATAL) << "unimplemented criterion";
+    }
+  } else {
+    std::unordered_map<std::string, std::string> cfg; // unused
+    W2lSerializer::load(reloadpath, cfg, network, criterion);
+  }
+  LOG_MASTER(INFO) << "[Network] " << network->prettyString();
+  LOG_MASTER(INFO) << "[Network Params: " << numTotalParams(network) << "]";
+  LOG_MASTER(INFO) << "[Criterion] " << criterion->prettyString();
+
+  std::shared_ptr<fl::LinSegCriterion> linseg;
+  if (FLAGS_linseg > 0) {
+    if (FLAGS_criterion != kAsgCriterion) {
+      LOG(FATAL) << "linseg may only be used with ASG criterion";
+    }
+    linseg = std::make_shared<fl::LinSegCriterion>(numClasses, scalemode);
+    linseg->setParams(criterion->param(0), 0);
+    LOG_MASTER(INFO) << "[Criterion] " << linseg->prettyString()
+                     << " (for first " << FLAGS_linseg << " epochs)";
+  }
+
+  /* ===================== Meters ===================== */
+  TrainMeters meters;
+  for (const auto& s : validsets) {
+    meters.valid[s] = EditDistMeters();
+  }
+
+  // best perf so far on valid datasets
+  std::unordered_map<std::string, double> validminerrs;
+  for (const auto& s : validsets) {
+    validminerrs[s] = DBL_MAX;
+  }
+
+  /* ===================== Logging ===================== */
+  std::ofstream perfile, logfile;
+  if (isMaster) {
+    dirCreate(runpath);
+    perfile.open(getRunFile("perf", runidx, runpath), std::ios::out);
+    logfile.open(getRunFile("log", runidx, runpath), std::ios::out);
+
+    // write header to perfile
+    auto msgp = getStatus(meters, 0, 0, 0, false, true, "\t").first;
+    print2file(perfile, "# " + msgp);
+    // write config
+    std::ofstream cfgfile(getRunFile("config", runidx, runpath));
+    cereal::JSONOutputArchive ar(cfgfile);
+    ar(CEREAL_NVP(config));
+  }
+
+  auto logStatus =
+      [&perfile, &logfile, isMaster](
+          TrainMeters& mtrs, int64_t epoch, double lr, double lrcrit) {
+        syncMeter(mtrs);
+
+        if (isMaster) {
+          auto msgl =
+              getStatus(mtrs, epoch, lr, lrcrit, true, false, " | ").second;
+          auto msgp = getStatus(mtrs, epoch, lr, lrcrit, false, true).second;
+          LOG_MASTER(INFO) << msgl;
+          print2file(logfile, msgl);
+          print2file(perfile, msgp);
+        }
+      };
+
+  auto saveModels = [&](int iter) {
+    if (isMaster) {
+      // Save last epoch
+      config[kEpoch] = std::to_string(iter);
+
+      std::string filename;
+      if (FLAGS_itersave) {
+        filename =
+            getRunFile(format("model_iter_%03d.bin", iter), runidx, runpath);
+      } else {
+        filename = getRunFile("model_last.bin", runidx, runpath);
+      }
+      // save last model
+      W2lSerializer::save(filename, config, network, criterion);
+
+      // save if better than ever for one valid
+      for (const auto& v : validminerrs) {
+        double verr = meters.valid[v.first].edit.value()[0];
+        if (verr < validminerrs[v.first]) {
+          validminerrs[v.first] = verr;
+          std::string cleaned_v = cleanFilepath(v.first);
+          std::string vfname =
+              getRunFile("model_" + cleaned_v + ".bin", runidx, runpath);
+          W2lSerializer::save(vfname, config, network, criterion);
+        }
+      }
+    }
+  };
+
+  /* ===================== Create Dataset ===================== */
+  auto createDataset = [worldRank, worldSize, &dicts](const std::string& path) {
+    std::shared_ptr<W2lDataset> ds;
+    if (FLAGS_everstoredb) {
+#ifdef BUILD_FB_DEPENDENCIES
+      W2lEverstoreDataset::init(); // Required for everstore client
+      ds = std::make_shared<W2lEverstoreDataset>(
+          path, dicts, FLAGS_batchsize, worldRank, worldSize, FLAGS_targettype);
+#else
+      LOG(FATAL) << "W2lEverstoreDataset not supported: "
+                 << "build with -DBUILD_FB_DEPENDENCIES";
+#endif
+    } else {
+      ds = std::make_shared<W2lNumberedFilesDataset>(
+          path, dicts, FLAGS_batchsize, worldRank, worldSize, FLAGS_datadir);
+    }
+    return ds;
+  };
+
+  auto trainds = createDataset(FLAGS_train);
+
+  if (FLAGS_noresample) {
+    LOG_MASTER(INFO) << "Shuffling trainset";
+    trainds->shuffle(FLAGS_seed);
+  }
+
+  std::unordered_map<std::string, std::shared_ptr<W2lDataset>> validds;
+  for (const auto& s : validsets) {
+    validds[s] = createDataset(s);
+  }
+
+  /* ===================== Hooks ===================== */
+  auto evalOutput = [&dicts, &criterion](
+                        const af::array& op,
+                        const af::array& target,
+                        fl::EditDistanceMeter& mtr) {
+    auto batchsz = op.dims(2);
+    for (int b = 0; b < batchsz; ++b) {
+      auto tgt = target(af::span, b);
+      auto viterbipath =
+          afToVector<int>(criterion->viterbiPath(op(af::span, af::span, b)));
+      auto tgtraw = afToVector<int>(tgt);
+
+      // Remove `-1`s appended to the target for batching (if any)
+      auto labellen = getTargetSize(tgtraw.data(), tgtraw.size());
+      tgtraw.resize(labellen);
+
+      // remap actual, predicted targets for evaluating edit distance error
+      if (dicts.find(kTargetIdx) == dicts.end()) {
+        LOG(FATAL) << "Dictionary not provided for target: " << kTargetIdx;
+      }
+      auto tgtDict = dicts.find(kTargetIdx)->second;
+
+      if (FLAGS_criterion == kCtcCriterion ||
+          FLAGS_criterion == kAsgCriterion) {
+        uniq(viterbipath);
+      }
+      if (FLAGS_criterion == kCtcCriterion) {
+        auto blankidx = tgtDict.getIndex(kBlankToken);
+        viterbipath.erase(
+            std::remove(viterbipath.begin(), viterbipath.end(), blankidx),
+            viterbipath.end());
+      }
+
+      remapLabels(viterbipath, tgtDict);
+      remapLabels(tgtraw, tgtDict);
+
+      mtr.add(
+          viterbipath.data(), tgtraw.data(), viterbipath.size(), tgtraw.size());
+    }
+  };
+
+  auto test = [&evalOutput](
+                  std::shared_ptr<fl::Module> ntwrk,
+                  std::shared_ptr<fl::Loss> crit,
+                  std::shared_ptr<W2lDataset> testds,
+                  EditDistMeters& mtrs) {
+    ntwrk->eval();
+    crit->eval();
+    mtrs.edit.reset();
+    mtrs.wordedit.reset();
+
+    for (auto& sample : *testds) {
+      auto output = ntwrk->forward(fl::input(sample[kInputIdx]));
+      evalOutput(output.array(), sample[kTargetIdx], mtrs.edit);
+    }
+  };
+
+  double gradNorm = 1.0 / (FLAGS_batchsize * worldSize);
+
+  auto train = [&meters,
+                &test,
+                &logStatus,
+                &saveModels,
+                &evalOutput,
+                &validds,
+                gradNorm,
+                startEpoch](
+                   std::shared_ptr<fl::Module> ntwrk,
+                   std::shared_ptr<fl::Loss> crit,
+                   std::shared_ptr<W2lDataset> trainset,
+                   fl::FirstOrderOptimizer& netopt,
+                   fl::FirstOrderOptimizer& critopt,
+                   bool clampCrit,
+                   int nepochs) {
+    fl::distributeModuleGrads(ntwrk, gradNorm);
+    fl::distributeModuleGrads(crit, gradNorm);
+
+    meters.loss.reset();
+    meters.train.edit.reset();
+    meters.train.wordedit.reset();
+
+    fl::allReduceParameters(ntwrk);
+    fl::allReduceParameters(crit);
+
+    auto resetTimeStatMeters = [&meters]() {
+      meters.runtime.reset();
+      meters.stats.reset();
+      meters.sampletimer.reset();
+      meters.fwdtimer.reset();
+      meters.critfwdtimer.reset();
+      meters.bwdtimer.reset();
+      meters.optimtimer.reset();
+      meters.timer.reset();
+    };
+    auto runValAndSaveModel = [&](int64_t epoch, double lr, double lrcrit) {
+      meters.runtime.stop();
+      meters.timer.stop();
+      meters.sampletimer.stop();
+      meters.fwdtimer.stop();
+      meters.critfwdtimer.stop();
+      meters.bwdtimer.stop();
+      meters.optimtimer.stop();
+
+      // valid
+      for (auto& vds : validds) {
+        test(ntwrk, crit, vds.second, meters.valid[vds.first]);
+      }
+
+      // print status
+      logStatus(meters, epoch, lr, lrcrit);
+      // save last and best models
+      saveModels(epoch);
+      // reset meters for next readings
+      meters.loss.reset();
+      meters.train.edit.reset();
+      meters.train.wordedit.reset();
+    };
+
+    int64_t curEpoch = startEpoch;
+    int64_t sampleIdx = 0;
+    while (curEpoch < nepochs) {
+      ++curEpoch;
+      ntwrk->train();
+      crit->train();
+      if (FLAGS_reportiters == 0) {
+        resetTimeStatMeters();
+      }
+      if (!FLAGS_noresample) {
+        LOG_MASTER(INFO) << "Shuffling trainset";
+        trainset->shuffle(curEpoch /* seed */);
+      }
+      af::sync();
+      meters.sampletimer.resume();
+      meters.runtime.resume();
+      meters.timer.resume();
+      LOG_MASTER(INFO) << "Epoch " << curEpoch << " started!";
+      for (auto& sample : *trainset) {
+        // meters
+        ++sampleIdx;
+        af::sync();
+        meters.timer.incUnit();
+        meters.sampletimer.stopAndIncUnit();
+        meters.stats.add(sample[kInputIdx], sample[kTargetIdx]);
+        if (af::anyTrue<bool>(af::isNaN(sample[kInputIdx])) ||
+            af::anyTrue<bool>(af::isNaN(sample[kTargetIdx]))) {
+          LOG(FATAL) << "Sample has NaN values";
+        }
+
+        // forward
+        meters.fwdtimer.resume();
+        auto output = ntwrk->forward(fl::input(sample[kInputIdx]));
+        af::sync();
+        meters.critfwdtimer.resume();
+        auto loss = crit->forward(output, fl::noGrad(sample[kTargetIdx]));
+        af::sync();
+        meters.fwdtimer.stopAndIncUnit();
+        meters.critfwdtimer.stopAndIncUnit();
+
+        if (af::anyTrue<bool>(af::isNaN(loss.array()))) {
+          LOG(FATAL) << "Loss has NaN values";
+        }
+        auto batchLoss = afToVector<float>(loss.array());
+        for (const auto lossval : batchLoss) {
+          meters.loss.add(lossval);
+        }
+
+        int64_t batchIdx = (sampleIdx - 1) % trainset->size();
+        int64_t globalBatchIdx = trainset->getGlobalBatchIdx(batchIdx);
+        if (globalBatchIdx % 100 < FLAGS_pcttraineval) {
+          evalOutput(output.array(), sample[kTargetIdx], meters.train.edit);
+        }
+
+        // backward
+        meters.bwdtimer.resume();
+        netopt.zeroGrad();
+        critopt.zeroGrad();
+        loss.backward();
+
+        af::sync();
+        meters.bwdtimer.stopAndIncUnit();
+        meters.optimtimer.resume();
+
+        if (FLAGS_maxgradnorm > 0) {
+          auto params = ntwrk->params();
+          if (clampCrit) {
+            auto critparams = crit->params();
+            params.insert(params.end(), critparams.begin(), critparams.end());
+          }
+          fl::clipGradNorm(params, FLAGS_maxgradnorm);
+        }
+        critopt.step();
+        netopt.step();
+        af::sync();
+        meters.optimtimer.stopAndIncUnit();
+        meters.sampletimer.resume();
+
+        if (FLAGS_reportiters > 0 && sampleIdx % FLAGS_reportiters == 0) {
+          runValAndSaveModel(curEpoch, netopt.getLr(), critopt.getLr());
+          resetTimeStatMeters();
+          ntwrk->train();
+          crit->train();
+          meters.sampletimer.resume();
+          meters.runtime.resume();
+          meters.timer.resume();
+        }
+      }
+      af::sync();
+      if (FLAGS_reportiters == 0) {
+        runValAndSaveModel(curEpoch, netopt.getLr(), critopt.getLr());
+      }
+      double lrScale = std::pow(FLAGS_gamma, curEpoch / FLAGS_stepsize);
+      netopt.setLr(lrScale * FLAGS_lr);
+      critopt.setLr(lrScale * FLAGS_lrcrit);
+    }
+  };
+
+  /* ===================== Train ===================== */
+  if (FLAGS_linseg > 0) {
+    auto linlr = FLAGS_linlr >= 0.0 ? FLAGS_linlr : FLAGS_lr;
+    auto linlrcrit = FLAGS_linlrcrit >= 0.0 ? FLAGS_linlrcrit : FLAGS_lrcrit;
+
+    auto linNetoptim = fl::SGDOptimizer(
+        network->params(), linlr, FLAGS_momentum, FLAGS_weightdecay);
+    auto linCritoptim = fl::SGDOptimizer(linseg->params(), linlrcrit, 0.0, 0.0);
+
+    train(
+        network,
+        linseg,
+        trainds,
+        linNetoptim,
+        linCritoptim,
+        false /* clampCrit */,
+        FLAGS_linseg);
+  }
+
+  // TODO - add support for other optimizers
+  auto netoptim = fl::SGDOptimizer(
+      network->params(), FLAGS_lr, FLAGS_momentum, FLAGS_weightdecay);
+  auto critoptim =
+      fl::SGDOptimizer(criterion->params(), FLAGS_lrcrit, 0.0, 0.0);
+
+  train(
+      network,
+      criterion,
+      trainds,
+      netoptim,
+      critoptim,
+      true /* clampCrit */,
+      FLAGS_iter);
+
+  LOG_MASTER(INFO) << "Finished training";
+
+  perfile.close();
+  logfile.close();
+
+  return 0;
+}
diff --git a/src/somef/test/test_data/repositories/wav2letter/docs/DecoderSketch.png b/src/somef/test/test_data/repositories/wav2letter/docs/DecoderSketch.png
new file mode 100644
index 00000000..494e8343
Binary files /dev/null and b/src/somef/test/test_data/repositories/wav2letter/docs/DecoderSketch.png differ
diff --git a/src/somef/test/test_data/repositories/wav2letter/docs/arch.md b/src/somef/test/test_data/repositories/wav2letter/docs/arch.md
new file mode 100644
index 00000000..7ea053d0
--- /dev/null
+++ b/src/somef/test/test_data/repositories/wav2letter/docs/arch.md
@@ -0,0 +1,68 @@
+# Writing Architecture Files
+
+wav2letter++ provides a simple way to create `fl::Sequential` module for the acoustic model from text files. These are specified using  the gflags `-arch` and `-archdir`.
+
+Example architecture file:
+```
+# Comments like this are ignored
+V -1 1 NFEAT 0
+C1 NFEAT 300 48 2 39
+R
+C1 300 300 32 1
+R
+RO 2 0 3 1
+L 300 NLABEL
+```
+
+While parsing, we ignore lines stating with `#` as comments. We also replace the following tokens `NFEAT` = input feature size (e.g. number of frequency bins), `NLABEL` = output size (e.g. number of grapheme tokens)
+
+The first token in each line represents a specific flashlight module followed by the specification of its parameters.
+
+Here, we describe how to specify different flashlight modules in the architecture files.
+
+**fl::Conv2D** `C2 [inputChannels] [outputChannels] [xFilterSz] [yFilterSz] [xStride] [yStride] [xPadding] [yPadding]`  
+
+*(Use padding `= -1` for `fl::PaddingMode::SAME`)* <br/>
+
+**fl::Linear** `L [inputChannels] [outputChannels]` <br/>
+
+**fl::BatchNorm** `BN [firstDim] [secondDim <OPTIONAL>] [thirdDim <OPTIONAL>]` <br/>
+
+**fl::LayerNorm** `LN [firstDim] [secondDim <OPTIONAL>] [thirdDim <OPTIONAL>]` <br/>
+
+**fl::WeightNorm** `WN [normDim] [Layer]` <br/>
+
+**fl::Dropout** `DO [dropProb]` <br/>
+
+**fl::Pool2D**
+   1. Average : `A [xFilterSz] [yFilterSz] [xStride] [yStride] [xPadding] [yPadding]`
+   1. Max : `M [xFilterSz] [yFilterSz] [xStride] [yStride] [xPadding] [yPadding]`
+
+*(Use padding `= -1` for `fl::PaddingMode::SAME`)* <br/>
+
+**fl::View** `V [firstDim] [secondDim] [thirdDim] [fourthDim]`
+
+*(Use `-1` to infer dimension, only one param can be a `-1`. Use `0` to use the corresponding input dimension.)* <br/>
+
+**fl::Reorder** `RO [firstDim] [secondDim] [thirdDim] [fourthDim]` <br/>
+
+**fl::ELU** `ELU` <br/>
+
+**fl::ReLU** `R`  <br/>
+
+**fl::PReLU** `PR [numElements <OPTIONAL>] [initValue <OPTIONAL>]`  <br/>
+
+**fl::Log** `LG` <br/>
+
+**fl::HardTanh** `HT`  <br/>
+
+**fl::Tanh** `T` <br/>
+
+**fl::GatedLinearUnit** `GLU [sliceDim]`  <br/>
+
+**fl::LogSoftmax** `LSM [normDim]` <br/>
+
+**fl::RNN**
+   1. RNN : `RNN [inputSize] [outputSize] [numLayers] [isBidirectional] [dropProb]`
+   1. GRU : `GRU [inputSize] [outputSize] [numLayers] [isBidirectional] [dropProb]`
+   1. LSTM : `LSTM [inputSize] [outputSize] [numLayers] [isBidirectional] [dropProb]`
diff --git a/src/somef/test/test_data/repositories/wav2letter/docs/decoder.md b/src/somef/test/test_data/repositories/wav2letter/docs/decoder.md
new file mode 100644
index 00000000..1b34f0dc
--- /dev/null
+++ b/src/somef/test/test_data/repositories/wav2letter/docs/decoder.md
@@ -0,0 +1,80 @@
+# Decoder
+![Decoder Diagram](DecoderSketch.png "Decoder Diagram")
+
+There are two components in decoding — test and decode.
+
+* The test stage is to compute basic statistics like letter error rate (LER) and word error rate (WER) given an acoustic model using the greedy best path from the Viterbi algorithm. It will also generate an `Emission Set` including the emission matrix as well as other target-related information for each sample, so that the `Emission Set` can be fed into decoder directly to generate transcripts without running feed forward for raw wave inputs again.
+* The decode stage focuses on minimizing the WER using a beam search decoder with a language model. It can take as input either an emission set generated from the test stage, which enables running hyper-parameter search in parallel, or an acoustic model to generate emissions at runtime, which may be more convenient when the dataset being decoded is small and a hyper-parameter sweep is not required.
+
+Aside from dataset and acoustic model, we also need two dictionaries as input.
+
+* Token dictionary: Same as the dictionary used to train the acoustic model and the order of tokens should also be identical. Each line in it is a unique token.
+* Word dictionary: All the candidate words the decoder refers to. Each line in it is a pair of word and its spelling split by a tab space, and tokens in the spelling represented by a string each are split by space (e.g. `apple	a p p l e`).
+
+
+### Building
+```
+buck build @mode/opt -c fbcode.platform=gcc-5-glibc-2.23 deeplearning/projects/wav2letter:test_cpp
+buck build @mode/opt -c fbcode.platform=gcc-5-glibc-2.23 deeplearning/projects/wav2letter:decode_cpp
+```
+
+### Running the `Test`
+Two dictionaries are specified through flags `tokens` and `lexicon`. We also have to set flag `am` and `emission_dir` to the path of the acoustic model and the directory where we want to save the emission set. Flags `datadir` and `test` are combined to specify the datasets we want to run experiment on. Note that we can test on more than 1 datasets, they have to be in the same `datadir` and are specified in `test` flag separated by a single comma in between.
+```
+<test_cpp_binary> \
+-tokens <path/to/tokens.txt> \
+-lexicon <path/to/words.txt> \
+-am <path/to/acoustic_model.bin> \
+-emission_dir <path/to/emission_dir/> \
+-datadir <path/to/dataset/> \
+-test <path/to/testset/> \
+-maxload -1 \
+-show
+```
+
+### Running the `Decode`
+Decoder can take either an acoustic model or an emission set as input, but not both. So only one of the flags `am` and `emission_dir` should be set. In general, flags across Decode and Test have similar functions. All the hyper-parameters are self-documented and can be set accordingly. Flag `sclite` specifies the path to save the logs, including *stdout* log and hypothesis and references in *sclite* format ([trn]( http://www1.icsi.berkeley.edu/Speech/docs/sctk-1.2/infmts.htm#trn_fmt_name_0)).
+
+#### Using acoustic model
+```
+<decode_cpp_binary> \
+-tokens <path/to/tokens.txt> \
+-lexicon <path/to/words.txt> \
+-am <path/to/acoustic_model.bin> \
+-lm <path/to/language_model.bin> \
+-datadir <path/to/dataset/> \
+-test <path/to/testset/> \
+-sclite <path/to/save/logs/> \
+-lmweight 4 \
+-wordscore 2.2 \
+-maxload 50 \
+-beamsize 2500 \
+-beamscore 100 \
+-silweight -1 \
+-nthread_decoder 8 \
+-smearing max \
+-show \
+-showletters
+```
+
+#### Using emission set
+```
+<decode_cpp_binary> \
+-tokens <path/to/tokens.txt> \
+-lexicon <path/to/words.txt> \
+-emission_dir <path/to/emission_dir/> \
+-lm <path/to/language_model.bin> \
+-datadir <path/to/dataset/> \
+-test <path/to/testset/> \
+-sclite <path/to/save/logs/> \
+-lmweight 4 \
+-wordscore 2.2 \
+-maxload 50 \
+-beamsize 2500 \
+-beamscore 100 \
+-silweight -1 \
+-nthread_decoder 8 \
+-smearing max \
+-show \
+-showletters
+```
diff --git a/src/somef/test/test_data/repositories/wav2letter/docs/installation.md b/src/somef/test/test_data/repositories/wav2letter/docs/installation.md
new file mode 100644
index 00000000..3e070b1e
--- /dev/null
+++ b/src/somef/test/test_data/repositories/wav2letter/docs/installation.md
@@ -0,0 +1,50 @@
+# Building wav2letter++
+
+## Build Requirements
+- A C++ compiler with good C++ 11 support (e.g. g++ >= 4.8)
+- [cmake](https://cmake.org/) — version 3.5.1 or later, make
+
+## Dependencies
+- [flashlight](https://github.com/facebookresearch/flashlight/) is required. flashlight _must_ be built with distributed training enabled.
+- [libsndfile](https://github.com/erikd/libsndfile) is required for loading audio. If using wav2letter++ with `flac` files, `libsndfile` [must be built](https://github.com/erikd/libsndfile#requirements) with `Ogg`, `Vorbis` and `FLAC` libraries.
+- [Intel's Math Kernel Library](https://software.intel.com/en-us/mkl) is required for featurization.
+- [FFTW](http://www.fftw.org/) is required for featurization.
+- [KenLM](https://github.com/kpu/kenlm) is required for the decoder. One of LZMA, BZip2, or Z is required for LM compression with KenLM.
+- [glags](https://github.com/gflags/gflags) is required.
+- [glog](https://github.com/google/glog) is required.
+- [Google Test](https://github.com/google/googletest) >= 1.8.0 is required if building tests.
+
+### Optional Dependencies
+- `flashlight` requires CUDA >= 9.2; if building wav2letter++ with the `CUDA` criterion backend, CUDA >= 9.2 is required. Using [CUDA 9.2](https://developer.nvidia.com/cuda-92-download-archive) is recommended.
+- If building with the `CPU` criteiron backend, wav2letter++ will try to compile with [OpenMP](https://www.openmp.org/), for better performance.
+
+## Build Options
+| Options           | Configuration     | Default Value |
+|-------------------|-------------------|---------------|
+| CRITERION_BACKEND | CUDA, CPU         | CUDA          |
+| BUILD_TESTS       | ON, OFF           | ON            |
+| CMAKE_BUILD_TYPE  | CMake build types | Debug         |
+
+## General Build Instructions
+First, clone the repository:
+```
+git clone --recursive https://github.com/facebookresearch/wav2letter.git
+```
+and follow the build instructions for your specific OS.
+
+There is no `install` procedure currently supported for wav2letter++. Building produces three binaries in the `build` directory:
+- `Train`: given a dataset of input audio and corresponding transcriptions in sub-word units (graphemes, phonemes, etc), trains the acoustic model.
+- `Test`: performs inference on a given dataset with an acoustic model.
+- `Decode`: given an acoustic model/pre-computed network emissions and a language model, computes the most likely sequence of words for a given dataset.
+
+### Building on Linux
+wav2letter++ has been tested on Ubuntu 16.04 and CentOS 7.5.
+
+Building on Linux is simple. Once you've downloaded wav2letter++ and built and installed the required dependencies:
+```
+# in your wav2letter++ directory
+mkdir -p build
+cd build
+cmake .. -DCMAKE_BUILD_TYPE=Release -DCRITERION_BACKEND=[backend]
+make -j4 # (or any number of threads)
+```
diff --git a/src/somef/test/test_data/repositories/wav2letter/docs/train.md b/src/somef/test/test_data/repositories/wav2letter/docs/train.md
new file mode 100644
index 00000000..4ec464b6
--- /dev/null
+++ b/src/somef/test/test_data/repositories/wav2letter/docs/train.md
@@ -0,0 +1,86 @@
+# Train
+
+At its simplest, training a model can be invoked with
+
+```
+<train_cpp_binary> train --flagsfile=<path_to_flags>
+```
+
+The flags to the train binary can be passed in a flagfile (see this example
+[flags file](../recipes/timit/configs/conv_relu/train.cfg)) or as flags on the
+command line:
+
+```
+<train_cpp_binary> [train|continue|fork] \
+--datadir <path/to/data/> \
+--tokensdir <path/to/tokens/file/> \
+--archdir <path/to/architecture/files/> \
+--rundir <path/to/save/models/> \
+--arch   <name_of_architecture.arch> \
+--train  <train/datasets/> \
+--valid  <validation/datasets> \
+--lr=0.0001 \
+--lrcrit=0.0001
+```
+
+### Modes
+
+Training supports three modes: 
+
+- `train` : Train a model from scratch on the given training data.
+- `continue` : Continue training a saved model. This can be used for example to
+  fine-tune with a smaller learning rate. The `continue` option makes a best
+  effort to resume training from the most recent checkpoint of a given model as
+  if there were no interruptions.
+- `fork` : Create and train a new model from a saved model. This can be used
+  for example to adapt a saved model to a new dataset.
+
+### Flags
+
+We give a short description of some of the more important flags here. A
+complete list of the flag definitions and short descriptions of their meaning
+can be found [here](../src/common/Defines.cpp).
+
+The `datadir` flag is the base path to where all the `train` and `valid`
+dataset directories live. Every `train` path will be prefixed by `datadir`.
+Multiple datasets can be passed to `train` and `valid` as a comma-separated
+list.
+
+Similarly, the `archdir` and `tokensdir` are (optional) base paths to where the
+`arch` and `token` files live. For example, the complete architecture file path
+will be `<archdir>/<arch>`.
+
+The `rundir` flag is the base directory where the model will be saved and the
+`runname` is the subdirectory that will be created to save the model and
+training logs. If `runname` is unspecified a directory name based on the date,
+time and user will be created.
+
+Most of the training hyperparameter flags have default values. Many of these
+you will not need to change. Some of the more important ones include:
+
+```
+- `lr` : The learning rate for the model parameters.
+- `lrcrit` : The learning rate for the criterion parameters.
+- `criterion` : Which criterion (e.g. loss function) to use. Options include `ctc`,
+  `asg` or `seq2seq`.
+- `batchsize` : The size of the minibatch to use per GPU.
+- `maxgradnorm` : Clip the norm of gradient of the model and criterion parameters
+  to this value. NB the norm is computed and clipped on the aggregated model
+  and criterion parameters.
+```
+
+
+## Distributed
+
+wav2letter++ supports distributed training on multiple GPUs out of the box. To
+run on multiple GPUs set pass the flag `-enable_distributed true` and run with
+MPI:
+
+```
+mpirun -n 8 <train_cpp_binary> [train|continue|fork] \
+-enable_distributed true \
+<... other flags ..>
+```
+
+The above command will run data parallel training with 8 processes (e.g. on 8
+GPUs).
diff --git a/src/somef/test/test_data/repositories/wav2letter/scripts/OptimizeSortParams.cpp b/src/somef/test/test_data/repositories/wav2letter/scripts/OptimizeSortParams.cpp
new file mode 100644
index 00000000..f63ff6d0
--- /dev/null
+++ b/src/somef/test/test_data/repositories/wav2letter/scripts/OptimizeSortParams.cpp
@@ -0,0 +1,156 @@
+/**
+ * Copyright (c) Facebook, Inc. and its affiliates.
+ * All rights reserved.
+ *
+ * This source code is licensed under the BSD-style license found in the
+ * LICENSE file in the root directory of this source tree.
+ */
+
+#include <utility>
+#include <vector>
+
+#include <gflags/gflags.h>
+#include <glog/logging.h>
+
+#include "common/Defines.h"
+#include "common/Utils.h"
+#include "data/NumberedFilesLoader.h"
+#include "data/Utils.h"
+#include "fb/W2lEverstoreDataset.h"
+#include "feature/Sound.h"
+
+using namespace w2l;
+
+std::pair<int, int> getPaddingStats(
+    int64_t batchsize,
+    const std::vector<int64_t>& sortedSampleIndices,
+    const std::vector<SpeechSampleMetaInfo>& samples) {
+  double input_pad = 0;
+  int output_pad = 0;
+  for (int i = 0; i < sortedSampleIndices.size() - batchsize + 1;
+       i += batchsize) {
+    auto& sample = samples[sortedSampleIndices[i]];
+
+    double input_min, input_max;
+    input_min = input_max = sample.audiolength();
+
+    int64_t output_min, output_max;
+    output_min = output_max = sample.reflength();
+
+    for (int j = 1; j < batchsize; j++) {
+      auto& s = samples[sortedSampleIndices[i + j]];
+      output_max = std::max(s.reflength(), output_max);
+      output_min = std::min(s.reflength(), output_min);
+      input_max = std::max(s.audiolength(), input_max);
+      input_min = std::min(s.audiolength(), input_min);
+    }
+    auto input_diff_frames = (input_max - input_min) / kFrameStrideMs;
+    input_pad += input_diff_frames * batchsize;
+    output_pad += (output_max - output_min) * batchsize;
+  }
+
+  return std::make_pair((int)input_pad, output_pad);
+}
+
+void logPaddingStats(
+    const std::string& sortfn,
+    const std::vector<SpeechSampleMetaInfo>& samples,
+    int batchsize,
+    int input_bin_size,
+    int output_bin_size) {
+  auto sortedindices =
+      sortSamples(samples, sortfn, input_bin_size, output_bin_size);
+  auto paddingStats = getPaddingStats(batchsize, sortedindices, samples);
+
+  LOG(INFO) << "Sort Function: " << sortfn
+            << ", Audio Padding: " << std::get<0>(paddingStats)
+            << ", Transcript Padding: " << std::get<1>(paddingStats)
+            << ", Input Bin: " << input_bin_size
+            << ", Output Bin: " << output_bin_size;
+}
+
+int findBestParams(std::function<int(int)> sorter) {
+  auto best = sorter(1);
+  auto best_bin_size = 1;
+  for (int i = 2; i < 200; i += 1) {
+    auto result = sorter(i);
+    if (result < best) {
+      best = result;
+      best_bin_size = i;
+    }
+  }
+  return best_bin_size;
+}
+
+std::vector<SpeechSampleMetaInfo> loadEverstore(const std::string& path) {
+  W2lEverstoreDataset::init(); // Required for everstore client
+
+  auto dict = makeDictionary();
+  LOG(INFO) << "Number of classes (network) = " << dict.indexSize();
+
+  DictionaryMap dicts;
+  dicts.insert({kTargetIdx, dict});
+  W2lEverstoreDataset ds(path, dicts, FLAGS_batchsize, 0, 1, FLAGS_targettype);
+
+  std::vector<SpeechSampleMetaInfo> samples;
+  for (size_t i = 0; i < ds.getSamples().size(); ++i) {
+    const auto& sample = ds.getSamples()[i];
+    samples.emplace_back(sample.durationMs, sample.ref.length(), i);
+  }
+  return samples;
+}
+
+std::vector<SpeechSampleMetaInfo> loadLocal(const std::string& path) {
+  TargetExtMap targetexts;
+  targetexts.insert({kTargetIdx, FLAGS_target});
+  NumberedFilesLoader loader(path, FLAGS_input, targetexts);
+  std::vector<SpeechSampleMetaInfo> samples;
+  for (int64_t i = 0; i < loader.size(); ++i) {
+    auto audiofile = loader.filename(i, FLAGS_input);
+    auto targetfile = loader.filename(i, FLAGS_target);
+    auto info = speech::loadSoundInfo(audiofile.c_str());
+    auto durationMs = ((double)info.frames / info.samplerate) * 1e3;
+    auto ref = loadTarget(targetfile);
+    samples.emplace_back(durationMs, ref.size(), i);
+  }
+  return samples;
+}
+
+int main(int argc, char** argv) {
+  google::InitGoogleLogging(argv[0]);
+  if (argc < 2) {
+    LOG(FATAL) << "Specify a data path.";
+  }
+
+  std::string path = argv[1];
+  gflags::ParseCommandLineFlags(&argc, &argv, false);
+  int batchsize = FLAGS_batchsize;
+
+  auto samples = FLAGS_everstoredb ? loadEverstore(path) : loadLocal(path);
+
+  logPaddingStats("input", samples, batchsize, 0, 0);
+
+  auto input_spiral_sorter = [&](int output_bin_size) {
+    auto stats = getPaddingStats(
+        batchsize,
+        sortSamples(samples, "input_spiral", 0, output_bin_size),
+        samples);
+    return std::get<0>(stats) + std::get<1>(stats);
+  };
+
+  int best_out_size = findBestParams(input_spiral_sorter);
+  logPaddingStats("input_spiral", samples, batchsize, 0, best_out_size);
+
+  auto output_spiral_sorter = [&](int input_bin_size) {
+    auto stats = getPaddingStats(
+        batchsize,
+        sortSamples(samples, "output_spiral", input_bin_size, 0),
+        samples);
+    return std::get<0>(stats) + std::get<1>(stats);
+  };
+
+  int best_in_size = findBestParams(output_spiral_sorter);
+  logPaddingStats("output_spiral", samples, batchsize, best_in_size, 0);
+
+  return 0;
+}
diff --git a/src/somef/test/test_data/repositories/wav2letter/scripts/Seq2SeqAnalysis.cpp b/src/somef/test/test_data/repositories/wav2letter/scripts/Seq2SeqAnalysis.cpp
new file mode 100644
index 00000000..83acc996
--- /dev/null
+++ b/src/somef/test/test_data/repositories/wav2letter/scripts/Seq2SeqAnalysis.cpp
@@ -0,0 +1,215 @@
+/**
+ * Copyright (c) Facebook, Inc. and its affiliates.
+ * All rights reserved.
+ *
+ * This source code is licensed under the BSD-style license found in the
+ * LICENSE file in the root directory of this source tree.
+ */
+
+#include <flashlight/flashlight.h>
+
+#include <gflags/gflags.h>
+#include <glog/logging.h>
+
+#include "common/Defines.h"
+#include "common/Transforms.h"
+#include "common/Utils.h"
+#include "criterion/criterion.h"
+#include "data/W2lNumberedFilesDataset.h"
+#include "module/module.h"
+#include "runtime/Serial.h"
+
+using namespace w2l;
+
+namespace w2l {
+
+DEFINE_bool(
+    viewtranscripts,
+    false,
+    "Log the Reference and Hypothesis transcripts.");
+
+DEFINE_string(attndir, "", "Directory for attention output.");
+
+DEFINE_int64(beamsz, 1, "Size of beam for beam search.");
+
+} // namespace w2l
+
+int main(int argc, char** argv) {
+  google::InitGoogleLogging(argv[0]);
+  std::string exec(argv[0]);
+
+  gflags::SetUsageMessage(
+      "Usage: \n " + exec +
+      " [model] [dataset], optional: --attndir=[directory]");
+
+  if (argc <= 2) {
+    LOG(FATAL) << gflags::ProgramUsage();
+  }
+
+  std::string reloadpath = argv[1];
+  std::string dataset = argv[2];
+  std::unordered_map<std::string, std::string> cfg;
+  std::shared_ptr<fl::Module> network;
+  std::shared_ptr<fl::SequenceCriterion> base_criterion;
+
+  W2lSerializer::load(reloadpath, cfg, network, base_criterion);
+  auto criterion =
+      std::dynamic_pointer_cast<fl::Seq2SeqCriterion>(base_criterion);
+
+  LOG(INFO) << "Parsing command line flags";
+  gflags::ParseCommandLineFlags(&argc, &argv, false);
+
+  auto flags = cfg.find(kGflags);
+  if (flags == cfg.end()) {
+    LOG(FATAL) << "Invalid config loaded from " << reloadpath;
+  }
+  LOG(INFO) << "Reading flags from config file " << reloadpath;
+  gflags::ReadFlagsFromString(flags->second, gflags::GetArgv0(), true);
+  LOG(INFO) << "Gflags after parsing \n" << serializeGflags("; ");
+
+  Dictionary dict = makeDictionary();
+
+  LOG(INFO) << "Number of classes (network) = " << dict.indexSize();
+
+  LOG(INFO) << "[network] " << network->prettyString();
+
+  af::setMemStepSize(FLAGS_memstepsize);
+  af::setSeed(FLAGS_seed);
+  std::string metername = FLAGS_target == "ltr" ? "LER: " : "PER: ";
+
+  fl::EditDistanceMeter cerMeter;
+  fl::EditDistanceMeter cerMeter_single;
+  fl::EditDistanceMeter cerBeamMeter;
+  fl::EditDistanceMeter werBeamMeter;
+  fl::EditDistanceMeter cerTfMeter;
+  fl::AverageValueMeter lossMeter;
+
+  DictionaryMap dicts;
+  dicts.insert({kTargetIdx, dict});
+
+  W2lNumberedFilesDataset testset(
+      dataset, dicts, FLAGS_batchsize, 0, 1, FLAGS_datadir);
+
+  auto toString = [&dict](const std::vector<int>& transcript) {
+    std::stringstream ss;
+    for (auto l : transcript) {
+      ss << dict.getToken(l);
+      if (FLAGS_target == "phn") {
+        ss << " ";
+      }
+    }
+    return ss.str();
+  };
+
+  network->eval();
+  criterion->eval();
+
+  int uid = 1;
+  for (auto& sample : testset) {
+    auto output = network->forward(fl::input(sample[kInputIdx]));
+    auto target = sample[kTargetIdx];
+    auto tfout =
+        std::get<0>(criterion->vectorizedDecoder(output, fl::noGrad(target)));
+    af::array maxv, argmaxid;
+    af::max(maxv, argmaxid, tfout.array(), 0);
+    argmaxid = argmaxid.as(s32);
+
+    auto loss = criterion->forward(output, fl::noGrad(target));
+    auto lossvec = afToVector<float>(loss.array());
+    for (int b = 0; b < output.dims(2); ++b) {
+      auto tgt = target(af::span, b);
+      auto teacherpath = afToVector<int>(argmaxid(af::span, af::span, b));
+      auto tgtraw = afToVector<int>(tgt);
+
+      fl::Variable attention;
+      af::array viterbipathArr;
+      std::tie(viterbipathArr, attention) = criterion->viterbiPathBase(
+          output.array()(af::span, af::span, b), (FLAGS_attndir != ""));
+
+      auto viterbipath = afToVector<int>(viterbipathArr);
+      auto beampath = viterbipath;
+      if (FLAGS_beamsz > 1) {
+        beampath = criterion->beamPath(
+            output.array()(af::span, af::span, b), FLAGS_beamsz);
+      }
+
+      // Truncate EOS tokens in teacherpath
+      auto eosloc = std::find(
+          teacherpath.begin(), teacherpath.end(), dict.getIndex(kEosToken));
+      teacherpath.resize(eosloc - teacherpath.begin());
+
+      remapLabels(beampath, dict);
+      remapLabels(viterbipath, dict);
+      remapLabels(tgtraw, dict);
+
+      cerMeter.add(viterbipath, tgtraw);
+      cerBeamMeter.add(beampath, tgtraw);
+      if (FLAGS_target == "ltr") {
+        auto beamwords = split("|", toString(beampath), true);
+        auto tgtwords = split("|", toString(tgtraw), true);
+        werBeamMeter.add(beamwords, tgtwords);
+      }
+      cerTfMeter.add(teacherpath, tgtraw);
+      lossMeter.add(lossvec[b]);
+
+      if (FLAGS_viewtranscripts) {
+        cerMeter_single.reset();
+        cerMeter_single.add(viterbipath, tgtraw);
+
+        std::cout << "UID: " << uid << ", " << metername
+                  << cerMeter_single.value()[0]
+                  << ", DEL: " << cerMeter_single.value()[2]
+                  << ", INS: " << cerMeter_single.value()[3]
+                  << ", SUB: " << cerMeter_single.value()[4] << std::endl;
+        std::cout << "REF     ";
+        std::cout << toString(tgtraw) << std::endl;
+
+        std::cout << "BEAM HYP  ";
+        std::cout << toString(beampath) << std::endl;
+
+        std::cout << "VP HYP  ";
+        std::cout << toString(viterbipath) << std::endl;
+
+        std::cout << "TF HYP  ";
+        std::cout << toString(teacherpath) << std::endl;
+
+        std::cout << "===============" << std::endl;
+      }
+
+      if (FLAGS_attndir != "") {
+        auto filename = FLAGS_attndir + "/" + std::to_string(uid) + "_attn.out";
+        std::string key(std::to_string(uid));
+        std::for_each(
+            viterbipath.begin(), viterbipath.end(), [&](const int& s) {
+              key += ("-" + dict.getToken(s));
+            });
+        key += "-<eos>";
+        af::saveArray(key.c_str(), attention.array(), filename.c_str(), false);
+      }
+      ++uid;
+    }
+  }
+
+  auto beamcer = cerBeamMeter.value()[0];
+  auto beamdel = cerBeamMeter.value()[2];
+  auto beamins = cerBeamMeter.value()[3];
+  auto beamsub = cerBeamMeter.value()[4];
+  auto viterbicer = cerMeter.value()[0];
+  auto viterbidel = cerMeter.value()[2];
+  auto viterbiins = cerMeter.value()[3];
+  auto viterbisub = cerMeter.value()[4];
+  auto tfcer = cerTfMeter.value()[0];
+  auto avgloss = lossMeter.value()[0];
+  LOG(INFO) << "Beam Search " << metername << beamcer << ", DEL: " << beamdel
+            << ", INS: " << beamins << ", SUB: " << beamsub;
+  LOG(INFO) << "Viterbi " << metername << viterbicer << ", DEL: " << viterbidel
+            << ", INS: " << viterbiins << ", SUB: " << viterbisub;
+  LOG(INFO) << "Teacher Forced " << metername << tfcer << ", Loss: " << avgloss;
+
+  if (FLAGS_target == "ltr") {
+    auto beamwer = werBeamMeter.value()[0];
+    LOG(INFO) << "Beam Search WER " << beamwer;
+  }
+
+  return 0;
+}
diff --git a/src/somef/test/test_data/repositories/wav2letter/scripts/arrayfire_parser.py b/src/somef/test/test_data/repositories/wav2letter/scripts/arrayfire_parser.py
new file mode 100644
index 00000000..238f1871
--- /dev/null
+++ b/src/somef/test/test_data/repositories/wav2letter/scripts/arrayfire_parser.py
@@ -0,0 +1,65 @@
+"""
+Copyright (c) Facebook, Inc. and its affiliates.
+All rights reserved.
+
+This source code is licensed under the BSD-style license found in the
+LICENSE file in the root directory of this source tree.
+"""
+
+from __future__ import absolute_import, division, print_function, unicode_literals
+import numpy as np
+
+
+def parser(filename):
+    # refer to af_dtype enum here
+    # http://arrayfire.org/docs/defines_8h.htm#a023d8ac325fb14f1712a52fb0940b1d5
+    dtype = {
+        0: 'f4',
+        1: 'c8',
+        2: 'f8',
+        3: 'c16',
+        4: 'bool',
+        5: 'i4',
+        6: 'u4',
+        7: 'u1',
+        8: 'i8',
+        9: 'u8',
+        10: 'i2',
+        11: 'u2',
+    }
+
+    with open(filename, "rb") as binary_file:
+        # refer to header definition at
+        # http://arrayfire.org/docs/group__stream__func__save.htm
+        binary_file.read(1)  # 1 byte for version
+        array_cnt_bytes = binary_file.read(4)
+        array_cnt = int.from_bytes(array_cnt_bytes, byteorder='little')
+
+        result = []
+        for _ac in range(array_cnt):
+            key_length_bytes = binary_file.read(4)
+            key_length = int.from_bytes(key_length_bytes, byteorder='little')
+            key_bytes = binary_file.read(key_length)
+            key = key_bytes.decode("utf-8")
+
+            binary_file.read(8)  # 8 bytes for offsets
+
+            af_dtype_byte = binary_file.read(1)
+            af_dtype = int.from_bytes(af_dtype_byte, byteorder='little')
+
+            dims = [1] * 4
+            for i in range(4):
+                dim_byte = binary_file.read(8)
+                dim = int.from_bytes(dim_byte, byteorder='little')
+                dims[i] = 1 if dim == 0 else dim
+
+            array = np.fromfile(
+                binary_file,
+                dtype=np.dtype(dtype[af_dtype]),
+                count=np.prod(np.array(dims)))
+
+            array = array.reshape(dims)
+
+            result.append((key, array))
+
+    return result
diff --git a/src/somef/test/test_data/repositories/wav2letter/scripts/fair_cluster/w2l_fair_cluster_build.sh b/src/somef/test/test_data/repositories/wav2letter/scripts/fair_cluster/w2l_fair_cluster_build.sh
new file mode 100644
index 00000000..6d40b832
--- /dev/null
+++ b/src/somef/test/test_data/repositories/wav2letter/scripts/fair_cluster/w2l_fair_cluster_build.sh
@@ -0,0 +1,37 @@
+#!/bin/bash
+
+## SLURM script for building wav2letter
+#SBATCH --job-name=wav2letter-build
+## %j is the job id, %u is the user id
+#SBATCH --output=/checkpoint/%u/jobs/wav2letter/build/wav2letter-build-%j.out
+#SBATCH --error=/checkpoint/%u/jobs/wav2letter/build/wav2letter-build-%j.err
+#SBATCH --partition=priority
+#SBATCH --comment="Quick building wav2letter"
+# Only one node and gpu needed for build
+#SBATCH --nodes=1
+# Use 1 attached Volta to more-easily auto-set nvcc flags
+#SBATCH --gres=gpu:volta:1
+#SBATCH --ntasks-per-node=1
+# Build on 32 cores
+#SBATCH --cpus-per-task 32
+#SBATCH --time=12:00:00
+
+### Set env vars
+# Start clean
+module purge
+# Load Dependencies
+module load cuda/9.2
+module load cudnn/v7.1-cuda.9.2
+module load NCCL/2.2.13-1-cuda.9.2
+module load mkl/2018.0.128
+module load openmpi/3.0.0/gcc.6.3.0
+module load kenlm/110617/gcc.6.3.0
+
+### Build
+# Assumes dependencies have been installed in $HOME/usr. This is REQUIRED
+# any dependencies which would otherwise install in locations like /usr
+# aren't accessible from learnfair machines. Any dynamic libraries
+# loaded at runtime must be in `/private/...` or another directory
+# mounted by/accessible from learnfair machines.
+export CMAKE_PREFIX_PATH="$HOME/usr"
+cd "$HOME/wav2letter/build/" && cmake .. && make -j32
diff --git a/src/somef/test/test_data/repositories/wav2letter/scripts/fair_cluster/w2l_fair_cluster_decode.sh b/src/somef/test/test_data/repositories/wav2letter/scripts/fair_cluster/w2l_fair_cluster_decode.sh
new file mode 100644
index 00000000..54f6d728
--- /dev/null
+++ b/src/somef/test/test_data/repositories/wav2letter/scripts/fair_cluster/w2l_fair_cluster_decode.sh
@@ -0,0 +1,34 @@
+#!/bin/bash
+
+## SLURM script for running decoding with wav2letter
+#SBATCH --job-name=wav2letter-decode
+## %j is the job id, %u is the user id
+#SBATCH --output=/checkpoint/%u/jobs/wav2letter/decode/wav2letter-decode-%j.out
+#SBATCH --error=/checkpoint/%u/jobs/wav2letter/decode/wav2letter-decode-%j.err
+## partition name
+#SBATCH --partition=scavenge
+## number of nodes
+# We only need one node, no gpus
+#SBATCH --nodes=1
+#SBATCH --gres=gpu:volta:0
+## number of tasks per node
+#SBATCH --ntasks-per-node=1
+#SBATCH --cpus-per-task 80
+#SBATCH --time=12:00:00
+
+# Runs on cpu
+~/wav2letter/build/Decoder \
+    -nthread_decoder 80 \
+    -show \
+    -criterion ctc \
+    -letters /private/home/jacobkahn/letter-dictionary.txt \
+    -words /private/home/jacobkahn/w2l-dict-openseq2seq-modified.txt \
+    -emission_dir /private/home/qiantong/dev_clean_emissions \
+    --logtostderr=1 \
+    -lm /private/home/qiantong/OpenSeq2Seq/language_model/4-gram.binary \
+    -smearing max \
+    -beamsize 100 \
+    -lmweight 3 \
+    -silweight -0.2 \
+    -wordscore 2.5 \
+    -beamscore 20 \
diff --git a/src/somef/test/test_data/repositories/wav2letter/scripts/fair_cluster/w2l_fair_cluster_distributed_training_benchmarks.sh b/src/somef/test/test_data/repositories/wav2letter/scripts/fair_cluster/w2l_fair_cluster_distributed_training_benchmarks.sh
new file mode 100644
index 00000000..894983ab
--- /dev/null
+++ b/src/somef/test/test_data/repositories/wav2letter/scripts/fair_cluster/w2l_fair_cluster_distributed_training_benchmarks.sh
@@ -0,0 +1,59 @@
+#!/bin/bash
+
+# Runs wav2letter in many different GPU/node configurations for benchmarking
+# distributed training/performance across many GPUs/nodes in the FAIR Cluster.
+# - Runs with 2^k GPUs for k in [1, 6]
+# - Assumes a SLURM-ready training script is in ~/wav2letter-train.sh
+# - Since single-GPU training requres a w2l flag changes, do manually
+
+# For a single node, run in 2, 4, 8 configurations
+single_node_gpu_quantities=("2" "4" "8")
+
+# Number of node configurations besides 1
+node_quantities=("2" "4" "8")
+
+# For SLURM sbatch, any variables passed from the command line
+# will take priority over variables defined by SBATCH
+# Flags that will not be overriden: partition, time, ntasks-per-node
+# see wav2letter-slurm.sh for defaults
+
+# Prefixes
+log_path_prefix=/checkpoint/%u/jobs/wav2letter/train-benchmark
+
+# Custom text for run name
+run_name_details=my-special-model
+
+# Run single node jobs
+# Iterate over num gpus
+for s in "${single_node_gpu_quantities[@]}"
+do
+    name="w2l-${run_name_details}-1-node-${s}-gpu"
+    echo "Starting job $name"
+    sbatch \
+        --job-name="${name}" \
+        --output="${log_path_prefix}/${name}-%j.out" \
+        --error="${log_path_prefix}/${name}-%j.err" \
+        --nodes=1 \
+        --gres="gpu:volta:${s}" \
+        --ntasks-per-node="${s}" \
+        --priority=priority \
+        --comment="Running benchmark - must avoid preemption " \
+        $HOME/wav2letter/scripts/fair_cluster/wav2letter-train.sh
+done
+
+# Run multi-node jobs. Always use 8 GPUs per node
+for s in "${node_quantities[@]}"
+do
+    name="w2l-${run_name_details}-${s}-node-8-gpu"
+    echo "Starting job $name"
+    sbatch \
+        --job-name="${name}" \
+        --output="${log_path_prefix}/${name}-%j.out" \
+        --error="${log_path_prefix}/${name}-%j.err" \
+        --nodes="${s}" \
+        --gres="gpu:volta:8" \
+        --ntasks-per-node="8" \
+        --partition=priority \
+        --comment="Running benchmark - must avoid preemption " \
+        $HOME/wav2letter/scripts/fair_cluster/wav2letter-train.sh
+done
diff --git a/src/somef/test/test_data/repositories/wav2letter/scripts/fair_cluster/w2l_fair_cluster_train.sh b/src/somef/test/test_data/repositories/wav2letter/scripts/fair_cluster/w2l_fair_cluster_train.sh
new file mode 100644
index 00000000..0be3fb58
--- /dev/null
+++ b/src/somef/test/test_data/repositories/wav2letter/scripts/fair_cluster/w2l_fair_cluster_train.sh
@@ -0,0 +1,48 @@
+#!/bin/bash
+
+## SLURM script for running distributed training with wav2letter
+#SBATCH --job-name=wav2letter-train
+## %j is the job id, %u is the user id
+#SBATCH --output=/checkpoint/%u/jobs/wav2letter/train/wav2letter-train-%j.out
+#SBATCH --error=/checkpoint/%u/jobs/wav2letter/train/wav2letter-train-%j.err
+## partition name
+#SBATCH --partition=scavenge
+## number of nodes
+#SBATCH --nodes=1
+# Example for 8 GPUs
+#SBATCH --gres=gpu:volta:8
+## number of tasks per node
+#SBATCH --ntasks-per-node=8
+#SBATCH --cpus-per-task 8
+#SBATCH --time=12:00:00
+# NOTE: if using fewer than 8 GPUs, the number of cpus-per-task MUST
+# be equal to the number of GPUs
+
+
+# Run job with mpi
+# A few notes:
+#   - Not using -np flag in MPI flag still signals the correct number of jobs
+#   - Keep num_nodes at 1, even if using multi-node, because MPI will do
+#     inter-node communication on the cluste
+mpirun ~/wav2letter/build/Train train \
+        -datadir /private/home/vineelkpratap/speech/pristine/ \
+        -train wsj2/nov93dev \
+        -valid wsj2/nov93dev \
+        -tokensdir /checkpoint/jacobkahn/wav2letter/examples/dict \
+        -tokens letters.lst \
+        -archdir /checkpoint/jacobkahn/wav2letter/archfiles \
+        -arch conv_20lyr_100mil \
+        -lr 0.0001 \
+        -target ltr \
+        -nthread 16 \
+        -criterion ctc \
+        -batchsize 4 \
+        -sqnorm \
+        -mfsc \
+        -onorm target \
+        --logtostderr=1 \
+        -rundir /checkpoint/jacobkahn/wav2letter/experiments/speech \
+        -runname "w2l-train-${SLURM_JOB_ID}"  \
+        --enable_distributed \
+        --num_nodes=1 \
+        -lsm
diff --git a/src/somef/test/test_data/repositories/wav2letter/scripts/imgcat b/src/somef/test/test_data/repositories/wav2letter/scripts/imgcat
new file mode 100755
index 00000000..19e843ff
--- /dev/null
+++ b/src/somef/test/test_data/repositories/wav2letter/scripts/imgcat
@@ -0,0 +1,118 @@
+# Copyright (c) 2017-present, Facebook, Inc.
+# All rights reserved.
+#
+# This source code is licensed under the BSD-style license found in the
+# LICENSE file in the root directory of this source tree.
+
+#!/bin/bash
+
+# tmux requires unrecognized OSC sequences to be wrapped with DCS tmux;
+# <sequence> ST, and for all ESCs in <sequence> to be replaced with ESC ESC. It
+# only accepts ESC backslash for ST.
+function print_osc() {
+    if [[ $TERM == screen* ]] ; then
+        printf "\033Ptmux;\033\033]"
+    else
+        printf "\033]"
+    fi
+}
+
+# More of the tmux workaround described above.
+function print_st() {
+    if [[ $TERM == screen* ]] ; then
+        printf "\a\033\\"
+    else
+        printf "\a"
+    fi
+}
+
+# print_image filename inline base64contents print_filename
+#   filename: Filename to convey to client
+#   inline: 0 or 1
+#   base64contents: Base64-encoded contents
+#   print_filename: If non-empty, print the filename
+#                   before outputting the image
+function print_image() {
+    print_osc
+    printf '1337;File='
+    if [[ -n "$1" ]]; then
+      printf 'name='`printf "%s" "$1" | base64`";"
+    fi
+
+    VERSION=$(base64 --version 2>&1)
+    if [[ "$VERSION" =~ fourmilab ]]; then
+      BASE64ARG=-d
+    elif [[ "$VERSION" =~ GNU ]]; then
+      BASE64ARG=-di
+    else
+      BASE64ARG=-D
+    fi
+
+    printf "%s" "$3" | base64 $BASE64ARG | wc -c | awk '{printf "size=%d",$1}'
+    printf ";inline=$2"
+    printf ":"
+    printf "%s" "$3"
+    print_st
+    printf '\n'
+    if [[ -n "$4" ]]; then
+      echo $1
+    fi
+}
+
+function error() {
+    echo "ERROR: $*" 1>&2
+}
+
+function show_help() {
+    echo "Usage: imgcat [-p] filename ..." 1>& 2
+    echo "   or: cat filename | imgcat" 1>& 2
+}
+
+## Main
+
+if [ -t 0 ]; then
+    has_stdin=f
+else
+    has_stdin=t
+fi
+
+# Show help if no arguments and no stdin.
+if [ $has_stdin = f -a $# -eq 0 ]; then
+    show_help
+    exit
+fi
+
+# Look for command line flags.
+while [ $# -gt 0 ]; do
+    case "$1" in
+    -h|--h|--help)
+        show_help
+        exit
+        ;;
+    -p|--p|--print)
+        print_filename=1
+        ;;
+    -*)
+        error "Unknown option flag: $1"
+        show_help
+        exit 1
+      ;;
+    *)
+        if [ -r "$1" ] ; then
+            has_stdin=f
+            print_image "$1" 1 "$(base64 < "$1")" "$print_filename"
+        else
+            error "imgcat: $1: No such file or directory"
+            exit 2
+        fi
+        ;;
+    esac
+    shift
+done
+
+# Read and print stdin
+if [ $has_stdin = t ]; then
+    print_image "" 1 "$(cat | base64)" ""
+fi
+
+exit 0
diff --git a/src/somef/test/test_data/repositories/wav2letter/scripts/launch_distributed_chronos.sh b/src/somef/test/test_data/repositories/wav2letter/scripts/launch_distributed_chronos.sh
new file mode 100755
index 00000000..d572f9f8
--- /dev/null
+++ b/src/somef/test/test_data/repositories/wav2letter/scripts/launch_distributed_chronos.sh
@@ -0,0 +1,60 @@
+#!/bin/bash
+function usage()
+{
+    echo "Usage: $0 [-n <NUM_NODES>][-p <NUM_PROCESSES_PER_NODE>][-f <COMMAND_FILE> | <COMMAND>]"
+}
+
+while getopts n:p:f: parm ; do
+case $parm in
+  n)
+    nnodes=$OPTARG
+    ;;
+  p)
+    nproc_per_node=$OPTARG
+    ;;
+  f)
+    cmd_file=$OPTARG
+    ;;
+  *)
+    usage
+    echo "Invalid argument"
+esac
+done
+
+shift $((OPTIND-1))
+
+if  [[ -z $cmd_file ]]; then
+  training_script=$*
+else
+  if [[ ! -f "$cmd_file" ]]; then
+    usage
+    echo "File not found : $cmd_file"
+    exit 1
+  fi
+  training_script=$(cat "$cmd_file")
+fi
+
+hostgroup=fblearner_ash_bigbasin_fair
+rndvpath="/mnt/vol/gfsai-east/ai-group/users/${USER}/chronos/rendezvous/"
+solibdir="/mnt/vol/gfsai-east/ai-group/users/${USER}"
+cpu_per_proc=6
+cpu_per_node=$((nproc_per_node*cpu_per_proc))
+
+if [ "$nnodes" -eq 1 ] && [ "$nproc_per_node" -eq 1 ]
+then
+  jobid=$(echo "AF_PATH=${solibdir} ${training_script}" \
+      | /usr/local/chronos/scripts/crun --hostgroup "${hostgroup}" --gpu "${nproc_per_node}" --cpu "${cpu_per_node}")
+elif [ "$nnodes" -eq 1 ]
+then
+  jobid=$(echo "AF_PATH=${solibdir} /usr/local/fbcode/gcc-5-glibc-2.23/bin/mpirun -n ${nproc_per_node} ${training_script} --enable_distributed" \
+      | /usr/local/chronos/scripts/crun --hostgroup "${hostgroup}" --gpu "${nproc_per_node}" --cpu "${cpu_per_node}")
+else
+  rand=$(echo -n $RANDOM | sha256sum | cut -c1-9)
+  echo "MultiNode training for wav2letter++ on AML cluster needs some adhoc modifications to the code. Please talk to @vineelkpratap."
+  exit 1
+  jobid=$(echo "AF_PATH=${solibdir} /usr/local/fbcode/gcc-5-glibc-2.23/bin/mpirun -n ${nproc_per_node} ${training_script} --enable_distributed --rndv_filepath=${rndvpath}/\${CHRONOS_JOB_ID}_${rand}" \
+      | /usr/local/chronos/scripts/crun --hostgroup "${hostgroup}" --gang-size "${nnodes}" --gang-rack-affinity --gpu "${nproc_per_node}" --cpu "${cpu_per_node}")
+fi
+
+echo "Job ID: ${jobid}"
+echo "  https://our.intern.facebook.com/intern/bunny/?x+${jobid}"
diff --git a/src/somef/test/test_data/repositories/wav2letter/scripts/tensorboard.py b/src/somef/test/test_data/repositories/wav2letter/scripts/tensorboard.py
new file mode 100644
index 00000000..e4fcfc03
--- /dev/null
+++ b/src/somef/test/test_data/repositories/wav2letter/scripts/tensorboard.py
@@ -0,0 +1,206 @@
+"""
+Copyright (c) Facebook, Inc. and its affiliates.
+All rights reserved.
+
+This source code is licensed under the BSD-style license found in the
+LICENSE file in the root directory of this source tree.
+"""
+
+"""
+Script to process the w2l logs to tensorboard summary format and store them in
+a new directory.
+
+Usage: tensorboard.py [-h] -i INDIR [-o OUTDIR] [-f FILTER] [-a APPEND]
+            [-n NTHREADS]
+
+Arguments:
+  -i INDIR, --indir INDIR
+                        Input directory containing experiments
+  -o OUTDIR, --outdir OUTDIR
+                        Output directory to log tensorboard summaries
+  -f FILTER, --filter FILTER
+                        Optional regex filter for filtering experiments by
+                        experiment directory name Ex: `.*timit.*`
+  -a APPEND, --append APPEND
+                        Optional comma-separated list of flags to be append to
+                        the directory name. Ex: `lr,train`
+                        Flag can be of the form `train='<some (regex)>'` too
+
+  -n NTHREADS, --nthreads NTHREADS
+                        Number threads to be used by ThreadPool
+"""
+
+from __future__ import absolute_import, division, print_function, unicode_literals
+
+import argparse
+import glob
+import os
+import re
+import string
+import sys
+import traceback
+from concurrent.futures import ALL_COMPLETED, ThreadPoolExecutor, wait
+
+import tensorflow as tf
+
+
+def parse_and_validate_args():
+    parser = argparse.ArgumentParser(description="Prepare tensorboard")
+    parser.add_argument(
+        "-i",
+        "--indir",
+        type=str,
+        required=True,
+        help="Input directory containing experiments",
+    )
+    parser.add_argument(
+        "-o",
+        "--outdir",
+        type=str,
+        default="/home/" + os.getenv("USER") + "/w2ltboard/",
+        help="Output directory to log tensorboard summaries",
+    )
+    parser.add_argument(
+        "-f",
+        "--filter",
+        type=str,
+        default=None,
+        help="Optional regex filter for filtering experiments by directory name "
+        " Ex: `.*timit.*`",
+    )
+    parser.add_argument(
+        "-a",
+        "--append",
+        type=str,
+        default=None,
+        help="Optional list of flags to be append to the directory name "
+        "(for filtering experiments in Tensorboard UI) Ex: `lr,train`. "
+        "Flag can be of the form `train='<some (regex)>'` in which case "
+        "the first match value will be used in place of the flag value",
+    )
+    parser.add_argument(
+        "-n",
+        "--nthreads",
+        type=int,
+        default=10,
+        help="Number threads to be used by ThreadPool",
+    )
+
+    args = parser.parse_args()
+
+    if not os.path.exists(args.indir):
+        print('Input directory"' + args.indir + "\" doesn't exist")
+        exit()
+
+    if not os.path.exists(args.outdir):
+        os.makedirs(args.outdir)
+
+    return args
+
+
+def is_valid_dir(dir, args):
+    if not os.path.isdir(os.path.join(args.indir, dir)):
+        return False
+    if not os.path.exists(os.path.join(args.indir, dir, "001_perf")):
+        return False
+    if not glob.glob(os.path.join(args.indir, dir, "001_config*")):
+        return False
+    if args.filter is not None and re.match(args.filter, dir) is None:
+        return False
+    return True
+
+
+def may_be_append_dir_name(dir, args):
+    if args.append is None:
+        return dir
+    flags = args.append.strip().split(",")
+    config_file = glob.glob(os.path.join(args.indir, dir, "001_config*"))[0]
+    valid_chars = "=-_.()%s%s" % (string.ascii_letters, string.digits)
+    with open(config_file) as file:
+        config = file.read()
+        for f in flags:
+            m = re.match("(.*)='(.*)'", f)
+            fre = None
+            if m:
+                f = m.group(1)
+                fre = m.group(2)
+            flg = re.search("(?<=\W)" + f + "[ ]*=.+?(?=(\\\\n--|,\n))", config)
+            if flg is not None:
+                lbl = flg.group(0)
+                if fre:
+                    lbl = re.search(fre, lbl)
+                    if lbl and len(lbl.groups()) > 0:
+                        lbl = f + "=" + lbl.group(1)
+                    else:
+                        lbl = f + "=UNK"
+                dir += ";" + "".join(c for c in lbl if c in valid_chars)
+    return dir
+
+
+def process_experiment(dir, args):
+    sys.stdout.write(".")
+    sys.stdout.flush()
+    if not is_valid_dir(dir, args):
+        return
+    basename = may_be_append_dir_name(os.path.basename(dir), args)
+    writer = tf.summary.FileWriter(
+        os.path.join(args.outdir, basename), graph=tf.get_default_graph()
+    )
+    perfid = 1
+    epoch = 1
+    while True:
+        perf_file = os.path.join(args.indir, dir, "{0:03d}_perf".format(perfid))
+        if not os.path.exists(perf_file):
+            break
+
+        with open(perf_file) as f:
+            content = f.readlines()
+        content = [x.strip() for x in content]
+
+        header = content[0].split()
+        summary_tags = {}
+        runtimeidx = -1
+        excluded_headers = ["filter.*", "epoch", "date", "time", "epoch", "#"]
+        for idx, h in enumerate(header):
+            if not any((re.match(eh, h)) for eh in excluded_headers):
+                if h == "runtime":
+                    h = h + " (minutes)"
+                    runtimeidx = idx - 1
+                summary_tags[idx - 1] = h
+        for idx in range(1, len(content)):
+            vals = content[idx].split()
+            if runtimeidx > 0:
+                hrs, min, sec = vals[runtimeidx].split(":")
+                mins = int(hrs) * 60 + int(min) + float(sec) / 60
+                vals[runtimeidx] = mins
+
+            summary = tf.Summary(
+                value=[
+                    tf.Summary.Value(tag=v, simple_value=float(vals[k]))
+                    for k, v in summary_tags.items()
+                ]
+            )
+            writer.add_summary(summary, epoch)
+            epoch = epoch + 1
+        perfid = perfid + 1
+    writer.flush()
+
+
+if __name__ == "__main__":
+    print("Tensorflow version:", tf.__version__)
+    args = parse_and_validate_args()
+    subdirs = os.listdir(args.indir)
+    futures = []
+    with ThreadPoolExecutor(max_workers=args.nthreads) as pool:
+        for s in subdirs:
+            futures.append(pool.submit(process_experiment, s, args))
+    wait(futures, return_when=ALL_COMPLETED)
+    # check for exceptions
+    for i in range(len(subdirs)):
+        try:
+            futures[i].result()
+        except Exception as e:
+            print("\nError while processing the experiment :", subdirs[i])
+            print(traceback.format_exc())
+            sys.exit(1)
+    print("\nDone!")
diff --git a/src/somef/test/test_data/repositories/wav2letter/scripts/wavdiff b/src/somef/test/test_data/repositories/wav2letter/scripts/wavdiff
new file mode 100755
index 00000000..7683d57e
--- /dev/null
+++ b/src/somef/test/test_data/repositories/wav2letter/scripts/wavdiff
@@ -0,0 +1,53 @@
+-- Copyright (c) 2017-present, Facebook, Inc.
+-- All rights reserved.
+
+-- This source code is licensed under the BSD-style license found in the
+-- LICENSE file in the root directory of this source tree.
+
+#!/bin/lua
+
+local ignore = {}
+for i=#arg,1,-1 do
+   if arg[i] == '-i' then
+      table.remove(arg, i)
+      local argignore = assert(table.remove(arg, i), '-i <comma separated list of fields to ignore> expected')
+      for k in argignore:gmatch('([^%s%,]+)') do
+         ignore[k] = true
+      end
+   end
+end
+
+local runs = {}
+
+for i, dir in ipairs(arg) do
+   local res, opt = pcall(dofile, dir .. "/001_config.lua")
+   if res then
+      opt = opt.opt
+   else
+      opt = nil
+   end
+   runs[i] = {dir=dir, opt=opt}
+end
+
+for i, run in ipairs(runs) do
+   local diffs = {}
+   if run.opt then
+      for k, v in pairs(run.opt) do
+         if not ignore[k] then
+            for j, orun in ipairs(runs) do
+               if orun.opt and orun.opt[k] ~= v then
+                  table.insert(diffs, string.format("%s=%s", k, tostring(v)))
+                  break
+               end
+            end
+         end
+      end
+   else
+      diffs = {"@FAILED"}
+   end
+   run.diffs = diffs
+end
+
+for i, run in ipairs(runs) do
+   print(run.dir, table.concat(run.diffs, ","))
+end
diff --git a/src/somef/test/test_data/repositories/wav2letter/scripts/wavextract b/src/somef/test/test_data/repositories/wav2letter/scripts/wavextract
new file mode 100755
index 00000000..91052032
--- /dev/null
+++ b/src/somef/test/test_data/repositories/wav2letter/scripts/wavextract
@@ -0,0 +1,180 @@
+-- Copyright (c) 2017-present, Facebook, Inc.
+-- All rights reserved.
+
+-- This source code is licensed under the BSD-style license found in the
+-- LICENSE file in the root directory of this source tree.
+
+#!/bin/lua
+
+local yt = 'dev'
+local filename -- filename for output
+local xtime = false -- time axis instead of iteration
+local dumb = false -- dumb terminal
+local legend = false -- is legend provided?
+
+for i=#arg,1,-1 do
+   if arg[i]:match('^%-') then
+      local opt = table.remove(arg, i)
+      if opt == '-y' then
+         yt = assert(table.remove(arg, i), '-y <string> expected')
+      elseif opt == '-o' then
+         filename = assert(table.remove(arg, i), '-o <filename> expected')
+      elseif opt == '-xtime' then
+         xtime = true
+      elseif opt == '-d' then
+         dumb = true
+      elseif opt == '-l' then
+         legend = true
+      else
+         error(string.format([[
+usage: %s [options] <dir1> <dir2> ...
+options:
+-y <number>  -- target to plot (%d)
+-o <string>  -- output filename (%s)
+-xtime       -- use training time for x axis (%s)
+-l           -- specifiy legends with <dir1> <legend1> <dir2> <legend2> ...
+]], arg[0], yt, filename, xtime))
+      end
+   end
+end
+
+local function split(str)
+   local s = {}
+   for x in str:gmatch('(%S+)') do
+      table.insert(s, x)
+   end
+   return s
+end
+
+local function totime(str)
+   local h, m, s = str:match('([^:]+):([^:]+):([^:]+)')
+   h, m, s = tonumber(h), tonumber(m), tonumber(s)
+   return h*3600+m*60+s
+end
+
+local function addplot(plots, dir, legend)
+   local x = {}
+   local y = {}
+   for i=1,999 do
+      local filename = string.format("%s/%03d_perf", dir, i)
+      local f = io.open(filename)
+      if not f then
+         break
+      end
+      f:close()
+      io.stderr:write(string.format('# processing: %s\n', filename))
+      local lines = io.lines(filename)
+      local header = lines()
+      if not header:match('^#') then
+         io.stderr:write(string.format('! warning: skipping directory <%s> (no header)\n', dir))
+         return
+      end
+      header = header:gsub('^#+', '')
+      local i = 0
+      local yc
+      local rc
+      for head in header:gmatch('(%S+)') do
+         i = i + 1
+         if head:find(yt) then
+            yc = i
+            break
+         end
+         if xtime and head == 'runtime' then
+            rc = i
+         end
+      end
+      if xtime and not rc then
+         io.stderr:write(string.format('! warning: skipping directory <%s> (no field <runtime>)\n', dir))
+      end
+      if not yc then
+         io.stderr:write(string.format('! warning: skipping directory <%s> (no field <%s>)\n', dir, yt))
+      end
+      for line in lines do
+         if not line:match('^#') then
+            local tokens = split(line)
+            table.insert(y, assert(tonumber(tokens[yc])))
+            if xtime then
+               table.insert(x, totime(tokens[rc]))
+            else
+               table.insert(x, #x+1)
+            end
+         end
+      end
+   end
+   if #y == 0 then
+      io.stderr:write(string.format('! warning: skipping directory <%s> (empty)\n', dir))
+   else
+      if xtime then
+         local cumsum = 0
+         for i=1,#x do
+            cumsum = cumsum + x[i]
+            x[i] = cumsum/3600
+         end
+      end
+      table.insert(plots, {legend, x, y})
+   end
+end
+
+local list = {}
+if legend then
+   if #arg % 2 ~= 0 then
+      error('<dir1> <legend1> <dir2> <legend2> ... expected')
+   end
+   for i=1,#arg/2 do
+      table.insert(
+         list,
+         {
+            dir = arg[2*(i-1)+1],
+            legend = arg[2*(i-1)+2]
+         }
+      )
+   end
+else
+   for i=1,#arg do
+      table.insert(
+         list,
+         {
+            dir = arg[i],
+            legend = arg[i]
+         }
+      )
+   end
+end
+
+local plots = {}
+for _, e in ipairs(list) do
+   addplot(plots, e.dir, e.legend)
+end
+
+local lmax = 0
+for i=1,#plots do
+   lmax = math.max(lmax, #plots[i][2])
+end
+
+for i=1,#plots do
+   if xtime then
+      io.write('time(h)\t')
+   else
+      io.write('iter\t')
+   end
+   io.write(plots[i][1])
+   if i < #plots then
+      io.write("\t")
+   end
+end
+print()
+for l=1,lmax do
+   for i=1,#plots do
+      if l <= #plots[i][2] then
+         io.write(string.format("%g", plots[i][2][l]))
+         io.write("\t")
+         io.write(string.format("%g", plots[i][3][l]))
+      else
+         io.write(".\t.")
+      end
+      if i ~= #plots then
+         io.write("\t")
+      end
+   end
+   print()
+end
diff --git a/src/somef/test/test_data/repositories/wav2letter/scripts/wavplot b/src/somef/test/test_data/repositories/wav2letter/scripts/wavplot
new file mode 100755
index 00000000..9e905b41
--- /dev/null
+++ b/src/somef/test/test_data/repositories/wav2letter/scripts/wavplot
@@ -0,0 +1,170 @@
+-- Copyright (c) 2017-present, Facebook, Inc.
+-- All rights reserved.
+
+-- This source code is licensed under the BSD-style license found in the
+-- LICENSE file in the root directory of this source tree.
+
+#!/bin/lua
+
+local xlabel
+local ylabel
+local title
+
+for i=#arg,1,-1 do
+   if arg[i] == '-x' then
+      table.remove(arg, i)
+      xlabel = assert(table.remove(arg, i), '-x <string> expected')
+   elseif arg[i] == '-y' then
+      table.remove(arg, i)
+      ylabel = assert(table.remove(arg, i), '-y <string> expected')
+   elseif arg[i] == '-t' then
+      table.remove(arg, i)
+      title = assert(table.remove(arg, i), '-t <string> expected')
+   end
+end
+if #arg > 0 then
+   error(string.format([[
+usage: %s [options]
+options:
+-x <string>  -- x legend
+-y <string>  -- y legend
+-t <string>  -- title
+]], arg[0]))
+end
+
+local config = [[
+
+red_000 = "#F9B7B0"
+red_025 = "#F97A6D"
+red_050 = "#E62B17"
+red_075 = "#8F463F"
+red_100 = "#6D0D03"
+
+blue_000 = "#A9BDE6"
+blue_025 = "#7297E6"
+blue_050 = "#1D4599"
+blue_075 = "#2F3F60"
+blue_100 = "#031A49"
+
+green_000 = "#A6EBB5"
+green_025 = "#67EB84"
+green_050 = "#11AD34"
+green_075 = "#2F6C3D"
+green_100 = "#025214"
+
+brown_000 = "#F9E0B0"
+brown_025 = "#F9C96D"
+brown_050 = "#E69F17"
+brown_075 = "#8F743F"
+brown_100 = "#6D4903"
+
+grid_color = "#d5e0c9"
+text_color = "#6a6a6a"
+
+my_font = "Verdana, 8"
+
+my_line_width = "2"
+my_axis_width = "1.5"
+my_ps = "1.2"
+
+set term pngcairo enhanced font my_font size 1024,768
+
+set pointsize my_ps
+
+set style line 1 linecolor rgbcolor blue_025 linewidth my_line_width pt 7
+set style line 2 linecolor rgbcolor green_025 linewidth my_line_width pt 5
+set style line 3 linecolor rgbcolor red_025 linewidth my_line_width pt 9
+set style line 4 linecolor rgbcolor brown_025 linewidth my_line_width pt 13
+set style line 5 linecolor rgbcolor blue_050 linewidth my_line_width pt 11
+set style line 6 linecolor rgbcolor green_050 linewidth my_line_width pt 7
+set style line 7 linecolor rgbcolor red_050 linewidth my_line_width pt 5
+set style line 8 linecolor rgbcolor brown_050 linewidth my_line_width pt 9
+set style line 9 linecolor rgbcolor blue_075 linewidth my_line_width pt 13
+set style line 10 linecolor rgbcolor green_075 linewidth my_line_width pt 11
+set style line 11 linecolor rgbcolor red_075 linewidth my_line_width pt 7
+set style line 12 linecolor rgbcolor brown_075 linewidth my_line_width pt 5
+set style line 13 linecolor rgbcolor blue_100 linewidth my_line_width pt 9
+set style line 14 linecolor rgbcolor green_100 linewidth my_line_width pt 13
+set style line 15 linecolor rgbcolor red_100 linewidth my_line_width pt 11
+set style line 16 linecolor rgbcolor brown_100 linewidth my_line_width pt 7
+set style line 17 linecolor rgbcolor "#224499" linewidth my_line_width pt 5
+
+set style increment user
+
+set xtics textcolor rgb text_color font my_font
+set ytics textcolor rgb text_color font my_font
+set ztics textcolor rgb text_color font my_font
+
+set label textcolor rgb text_color font my_font
+
+set border 31 lw my_axis_width lc rgb text_color
+
+set key width 1 height 1 noenhanced spacing 1
+
+]]
+
+local data = io.read('*all')
+local header = data:match('([^\n]+)')
+local tmpfname = os.tmpname()
+local f = io.open(tmpfname, "w")
+f:write(data)
+f:close()
+
+local plots = {}
+local i = 0
+for xt, yt in header:gmatch('(%S+)%s+(%S+)') do
+   yt = yt:gsub("_", "\\_")
+   io.stderr:write(yt)
+   io.stderr:write('\n')
+   i = i + 1
+   table.insert(
+      plots,
+      string.format(
+         '"%s" using %d:%d with linespoints title "%s"',
+         tmpfname,
+         (i-1)*2+1,
+         (i-1)*2+2, yt
+      )
+   )
+end
+
+local plot = {}
+if xlabel then
+   table.insert(
+      plot,
+      string.format(
+         'set xlabel "%s" textcolor rgb text_color font my_font',
+         xlabel
+      )
+   )
+end
+if ylabel then
+   table.insert(
+      plot,
+      string.format(
+         'set ylabel "%s" textcolor rgb text_color font my_font',
+         ylabel
+      )
+   )
+end
+if title then
+   table.insert(
+      plot,
+      string.format(
+         'set title "%s" textcolor rgb text_color font my_font',
+         title
+      )
+   )
+end
+table.insert(
+   plot,
+   string.format("plot %s\n", table.concat(plots, ", "))
+)
+plot = table.concat(plot, "\n")
+
+local f = io.popen('gnuplot', 'w')
+f:write(config)
+f:write(plot)
+f:close()
+
+os.execute(string.format('rm -f "%s"', tmpfname))
diff --git a/src/somef/test/test_data/repositories/wav2letter/scripts/wavstatus b/src/somef/test/test_data/repositories/wav2letter/scripts/wavstatus
new file mode 100755
index 00000000..b5e5fb45
--- /dev/null
+++ b/src/somef/test/test_data/repositories/wav2letter/scripts/wavstatus
@@ -0,0 +1,94 @@
+-- Copyright (c) 2017-present, Facebook, Inc.
+-- All rights reserved.
+
+-- This source code is licensed under the BSD-style license found in the
+-- LICENSE file in the root directory of this source tree.
+
+#!/bin/lua
+
+local id2run = {}
+local runs = {}
+
+function dir2id(dir)
+   dir = dir:gsub('/+$', '')
+   return assert(dir:match('([^/]+)$'):match('^(%d+)'), 'could not infer id')
+end
+
+for _, dir in ipairs(arg) do
+   local id = dir2id(dir)
+   id2run[id] = id2run[id] or {}
+   local run = id2run[id]
+   run.dir = dir
+   run.id = id
+
+   local runidx
+   for i=1,999 do
+      local fn = string.format("%s/%03d_config.lua", dir, i)
+      local f = io.open(fn)
+      if f then
+         f:close()
+         -- did this continued/forked something else?
+         local config = dofile(string.format("%s/%03d_config.lua", dir, i))
+         if config.reload then
+            local idr = dir2id(config.reload:gsub('[^/]+$', ''))
+            id2run[idr] = id2run[idr] or {}
+            if config.command == '--continue' then
+               id2run[idr].continued = true
+            elseif config.command == '--fork' then
+               id2run[idr].forked = true
+            else
+               error(string.format('unknown command <%s>', config.command))
+            end
+         end
+      else
+         runidx = i-1
+         break
+      end
+   end
+
+   -- is running?
+   -- this is slow:
+   -- local f = io.popen(string.format("/usr/local/chronos/scripts/cinfo --jobState %s", id))
+   -- local status = f:read('*l')
+   -- f:close()
+   -- print(string.format("status <%s>", status or "NONE"))
+
+   local fn = string.format("%s/heartbeat", dir)
+   local f = io.open(fn)
+   if f then
+      f:close()
+      local f = io.popen(string.format('stat -c %%Y "%s"', fn))
+      local ts = assert(tonumber(f:read('*all')), 'could not stat file')
+      f:close()
+      -- make sure this is not just a continuation by checking ids match
+      local config = dofile(string.format("%s/%03d_config.lua", dir, runidx))
+      if config.opt.runname == id then
+         local rts = os.date("%s")
+         if rts - ts > 900 then
+            run.finished = true
+         else
+            run.running = true
+         end
+      else
+         run.finished = true
+      end
+   else
+      run.failed = true
+   end
+   table.insert(runs, run)
+end
+
+for _, run in ipairs(runs) do
+   print(
+      run.dir,
+      run.id,
+      string.format(
+         "%s%s%s%s%s",
+         run.running and "@RUNNING" or "",
+         run.finished and "@FINISHED" or "",
+         run.failed and "@FAILED" or "",
+         run.forked and "@FORKED" or "",
+         run.continued and "@CONTINUED" or ""
+      )
+   )
+end
diff --git a/src/somef/test/test_data/test_465.md b/src/somef/test/test_data/test_465.md
new file mode 100644
index 00000000..71f92e1d
--- /dev/null
+++ b/src/somef/test/test_data/test_465.md
@@ -0,0 +1,388 @@
+# modelcatalog-api
+This is the API of the Software Description Ontology at [https://w3id.org/okn/o/sdm](https://w3id.org/okn/o/sdm)
+
+This Python package is automatically generated by the [OpenAPI Generator](https://openapi-generator.tech) project:
+
+- API version: v1.8.0
+- Package version: 8.0.0
+- Build package: org.openapitools.codegen.languages.PythonClientCodegen
+
+## Requirements.
+
+Python 2.7 and 3.4+
+
+## Installation & Usage
+### pip install
+
+If the python package is hosted on a repository, you can install directly using:
+
+```sh
+pip install git+https://github.com/mintproject/model-catalog-python-api-client.git
+```
+(you may need to run `pip` with root permission: `sudo pip install git+https://github.com/mintproject/model-catalog-python-api-client.git`)
+
+Then import the package:
+```python
+import modelcatalog 
+```
+
+### Setuptools
+
+Install via [Setuptools](http://pypi.python.org/pypi/setuptools).
+
+```sh
+python setup.py install --user
+```
+(or `sudo python setup.py install` to install the package for all users)
+
+Then import the package:
+```python
+import modelcatalog
+```
+
+## Getting Started
+
+Please follow the [installation procedure](#installation--usage) and then run the following:
+
+```python
+from __future__ import print_function
+import time
+import modelcatalog
+from modelcatalog.rest import ApiException
+from pprint import pprint
+
+
+# Defining host is optional and default to https://api.models.mint.isi.edu/v1.8.0
+configuration.host = "https://api.models.mint.isi.edu/v1.8.0"
+# Create an instance of the API class
+api_instance = modelcatalog.CatalogIdentifierApi(modelcatalog.ApiClient(configuration))
+username = 'username_example' # str | Name of the user graph to query (optional)
+label = 'label_example' # str | Filter by label (optional)
+page = 1 # int | Page number (optional) (default to 1)
+per_page = 100 # int | Items per page (optional) (default to 100)
+
+try:
+    # List all instances of CatalogIdentifier
+    api_response = api_instance.catalogidentifiers_get(username=username, label=label, page=page, per_page=per_page)
+    pprint(api_response)
+except ApiException as e:
+    print("Exception when calling CatalogIdentifierApi->catalogidentifiers_get: %s\n" % e)
+
+```
+
+## Documentation for API Endpoints
+
+All URIs are relative to *https://api.models.mint.isi.edu/v1.8.0*
+
+Class | Method | HTTP request | Description
+------------ | ------------- | ------------- | -------------
+*CatalogIdentifierApi* | [**catalogidentifiers_get**](docs/endpoints/CatalogIdentifierApi.md#catalogidentifiers_get) | **GET** /catalogidentifiers | List all instances of CatalogIdentifier
+*CatalogIdentifierApi* | [**catalogidentifiers_id_delete**](docs/endpoints/CatalogIdentifierApi.md#catalogidentifiers_id_delete) | **DELETE** /catalogidentifiers/{id} | Delete an existing CatalogIdentifier
+*CatalogIdentifierApi* | [**catalogidentifiers_id_get**](docs/endpoints/CatalogIdentifierApi.md#catalogidentifiers_id_get) | **GET** /catalogidentifiers/{id} | Get a single CatalogIdentifier by its id
+*CatalogIdentifierApi* | [**catalogidentifiers_id_put**](docs/endpoints/CatalogIdentifierApi.md#catalogidentifiers_id_put) | **PUT** /catalogidentifiers/{id} | Update an existing CatalogIdentifier
+*CatalogIdentifierApi* | [**catalogidentifiers_post**](docs/endpoints/CatalogIdentifierApi.md#catalogidentifiers_post) | **POST** /catalogidentifiers | Create one CatalogIdentifier
+*CausalDiagramApi* | [**causaldiagrams_get**](docs/endpoints/CausalDiagramApi.md#causaldiagrams_get) | **GET** /causaldiagrams | List all instances of CausalDiagram
+*CausalDiagramApi* | [**causaldiagrams_id_delete**](docs/endpoints/CausalDiagramApi.md#causaldiagrams_id_delete) | **DELETE** /causaldiagrams/{id} | Delete an existing CausalDiagram
+*CausalDiagramApi* | [**causaldiagrams_id_get**](docs/endpoints/CausalDiagramApi.md#causaldiagrams_id_get) | **GET** /causaldiagrams/{id} | Get a single CausalDiagram by its id
+*CausalDiagramApi* | [**causaldiagrams_id_put**](docs/endpoints/CausalDiagramApi.md#causaldiagrams_id_put) | **PUT** /causaldiagrams/{id} | Update an existing CausalDiagram
+*CausalDiagramApi* | [**causaldiagrams_post**](docs/endpoints/CausalDiagramApi.md#causaldiagrams_post) | **POST** /causaldiagrams | Create one CausalDiagram
+*ConfigurationSetupApi* | [**configurationsetups_get**](docs/endpoints/ConfigurationSetupApi.md#configurationsetups_get) | **GET** /configurationsetups | List all instances of ConfigurationSetup
+*ConfigurationSetupApi* | [**configurationsetups_id_delete**](docs/endpoints/ConfigurationSetupApi.md#configurationsetups_id_delete) | **DELETE** /configurationsetups/{id} | Delete an existing ConfigurationSetup
+*ConfigurationSetupApi* | [**configurationsetups_id_get**](docs/endpoints/ConfigurationSetupApi.md#configurationsetups_id_get) | **GET** /configurationsetups/{id} | Get a single ConfigurationSetup by its id
+*ConfigurationSetupApi* | [**configurationsetups_id_put**](docs/endpoints/ConfigurationSetupApi.md#configurationsetups_id_put) | **PUT** /configurationsetups/{id} | Update an existing ConfigurationSetup
+*ConfigurationSetupApi* | [**configurationsetups_post**](docs/endpoints/ConfigurationSetupApi.md#configurationsetups_post) | **POST** /configurationsetups | Create one ConfigurationSetup
+*ConfigurationSetupApi* | [**custom_configurationsetups_id_get**](docs/endpoints/ConfigurationSetupApi.md#custom_configurationsetups_id_get) | **GET** /custom/configurationsetups/{id} | Get a ModelConfigurationSetup
+*ConstraintApi* | [**constraints_get**](docs/endpoints/ConstraintApi.md#constraints_get) | **GET** /constraints | List all instances of Constraint
+*ConstraintApi* | [**constraints_id_delete**](docs/endpoints/ConstraintApi.md#constraints_id_delete) | **DELETE** /constraints/{id} | Delete an existing Constraint
+*ConstraintApi* | [**constraints_id_get**](docs/endpoints/ConstraintApi.md#constraints_id_get) | **GET** /constraints/{id} | Get a single Constraint by its id
+*ConstraintApi* | [**constraints_id_put**](docs/endpoints/ConstraintApi.md#constraints_id_put) | **PUT** /constraints/{id} | Update an existing Constraint
+*ConstraintApi* | [**constraints_post**](docs/endpoints/ConstraintApi.md#constraints_post) | **POST** /constraints | Create one Constraint
+*CoupledModelApi* | [**coupledmodels_get**](docs/endpoints/CoupledModelApi.md#coupledmodels_get) | **GET** /coupledmodels | List all instances of CoupledModel
+*CoupledModelApi* | [**coupledmodels_id_delete**](docs/endpoints/CoupledModelApi.md#coupledmodels_id_delete) | **DELETE** /coupledmodels/{id} | Delete an existing CoupledModel
+*CoupledModelApi* | [**coupledmodels_id_get**](docs/endpoints/CoupledModelApi.md#coupledmodels_id_get) | **GET** /coupledmodels/{id} | Get a single CoupledModel by its id
+*CoupledModelApi* | [**coupledmodels_id_put**](docs/endpoints/CoupledModelApi.md#coupledmodels_id_put) | **PUT** /coupledmodels/{id} | Update an existing CoupledModel
+*CoupledModelApi* | [**coupledmodels_post**](docs/endpoints/CoupledModelApi.md#coupledmodels_post) | **POST** /coupledmodels | Create one CoupledModel
+*DataTransformationApi* | [**custom_datasetspecifications_id_datatransformations_get**](docs/endpoints/DataTransformationApi.md#custom_datasetspecifications_id_datatransformations_get) | **GET** /custom/datasetspecifications/{id}/datatransformations | Gets a list of data transformations related a dataset
+*DataTransformationApi* | [**datatransformations_get**](docs/endpoints/DataTransformationApi.md#datatransformations_get) | **GET** /datatransformations | List all instances of DataTransformation
+*DataTransformationApi* | [**datatransformations_id_delete**](docs/endpoints/DataTransformationApi.md#datatransformations_id_delete) | **DELETE** /datatransformations/{id} | Delete an existing DataTransformation
+*DataTransformationApi* | [**datatransformations_id_get**](docs/endpoints/DataTransformationApi.md#datatransformations_id_get) | **GET** /datatransformations/{id} | Get a single DataTransformation by its id
+*DataTransformationApi* | [**datatransformations_id_put**](docs/endpoints/DataTransformationApi.md#datatransformations_id_put) | **PUT** /datatransformations/{id} | Update an existing DataTransformation
+*DataTransformationApi* | [**datatransformations_post**](docs/endpoints/DataTransformationApi.md#datatransformations_post) | **POST** /datatransformations | Create one DataTransformation
+*DataTransformationSetupApi* | [**datatransformationsetups_get**](docs/endpoints/DataTransformationSetupApi.md#datatransformationsetups_get) | **GET** /datatransformationsetups | List all instances of DataTransformationSetup
+*DataTransformationSetupApi* | [**datatransformationsetups_id_delete**](docs/endpoints/DataTransformationSetupApi.md#datatransformationsetups_id_delete) | **DELETE** /datatransformationsetups/{id} | Delete an existing DataTransformationSetup
+*DataTransformationSetupApi* | [**datatransformationsetups_id_get**](docs/endpoints/DataTransformationSetupApi.md#datatransformationsetups_id_get) | **GET** /datatransformationsetups/{id} | Get a single DataTransformationSetup by its id
+*DataTransformationSetupApi* | [**datatransformationsetups_id_put**](docs/endpoints/DataTransformationSetupApi.md#datatransformationsetups_id_put) | **PUT** /datatransformationsetups/{id} | Update an existing DataTransformationSetup
+*DataTransformationSetupApi* | [**datatransformationsetups_post**](docs/endpoints/DataTransformationSetupApi.md#datatransformationsetups_post) | **POST** /datatransformationsetups | Create one DataTransformationSetup
+*DatasetSpecificationApi* | [**custom_configuration_id_inputs_get**](docs/endpoints/DatasetSpecificationApi.md#custom_configuration_id_inputs_get) | **GET** /custom/configuration/{id}/inputs | Gets all inputs of a configuration
+*DatasetSpecificationApi* | [**custom_datasetspecifications_get**](docs/endpoints/DatasetSpecificationApi.md#custom_datasetspecifications_get) | **GET** /custom/datasetspecifications | Gets all inputs of a configuration
+*DatasetSpecificationApi* | [**datasetspecifications_get**](docs/endpoints/DatasetSpecificationApi.md#datasetspecifications_get) | **GET** /datasetspecifications | List all instances of DatasetSpecification
+*DatasetSpecificationApi* | [**datasetspecifications_id_delete**](docs/endpoints/DatasetSpecificationApi.md#datasetspecifications_id_delete) | **DELETE** /datasetspecifications/{id} | Delete an existing DatasetSpecification
+*DatasetSpecificationApi* | [**datasetspecifications_id_get**](docs/endpoints/DatasetSpecificationApi.md#datasetspecifications_id_get) | **GET** /datasetspecifications/{id} | Get a single DatasetSpecification by its id
+*DatasetSpecificationApi* | [**datasetspecifications_id_put**](docs/endpoints/DatasetSpecificationApi.md#datasetspecifications_id_put) | **PUT** /datasetspecifications/{id} | Update an existing DatasetSpecification
+*DatasetSpecificationApi* | [**datasetspecifications_post**](docs/endpoints/DatasetSpecificationApi.md#datasetspecifications_post) | **POST** /datasetspecifications | Create one DatasetSpecification
+*EmpiricalModelApi* | [**empiricalmodels_get**](docs/endpoints/EmpiricalModelApi.md#empiricalmodels_get) | **GET** /empiricalmodels | List all instances of EmpiricalModel
+*EmpiricalModelApi* | [**empiricalmodels_id_delete**](docs/endpoints/EmpiricalModelApi.md#empiricalmodels_id_delete) | **DELETE** /empiricalmodels/{id} | Delete an existing EmpiricalModel
+*EmpiricalModelApi* | [**empiricalmodels_id_get**](docs/endpoints/EmpiricalModelApi.md#empiricalmodels_id_get) | **GET** /empiricalmodels/{id} | Get a single EmpiricalModel by its id
+*EmpiricalModelApi* | [**empiricalmodels_id_put**](docs/endpoints/EmpiricalModelApi.md#empiricalmodels_id_put) | **PUT** /empiricalmodels/{id} | Update an existing EmpiricalModel
+*EmpiricalModelApi* | [**empiricalmodels_post**](docs/endpoints/EmpiricalModelApi.md#empiricalmodels_post) | **POST** /empiricalmodels | Create one EmpiricalModel
+*EmulatorApi* | [**emulators_get**](docs/endpoints/EmulatorApi.md#emulators_get) | **GET** /emulators | List all instances of Emulator
+*EmulatorApi* | [**emulators_id_delete**](docs/endpoints/EmulatorApi.md#emulators_id_delete) | **DELETE** /emulators/{id} | Delete an existing Emulator
+*EmulatorApi* | [**emulators_id_get**](docs/endpoints/EmulatorApi.md#emulators_id_get) | **GET** /emulators/{id} | Get a single Emulator by its id
+*EmulatorApi* | [**emulators_id_put**](docs/endpoints/EmulatorApi.md#emulators_id_put) | **PUT** /emulators/{id} | Update an existing Emulator
+*EmulatorApi* | [**emulators_post**](docs/endpoints/EmulatorApi.md#emulators_post) | **POST** /emulators | Create one Emulator
+*EquationApi* | [**equations_get**](docs/endpoints/EquationApi.md#equations_get) | **GET** /equations | List all instances of Equation
+*EquationApi* | [**equations_id_delete**](docs/endpoints/EquationApi.md#equations_id_delete) | **DELETE** /equations/{id} | Delete an existing Equation
+*EquationApi* | [**equations_id_get**](docs/endpoints/EquationApi.md#equations_id_get) | **GET** /equations/{id} | Get a single Equation by its id
+*EquationApi* | [**equations_id_put**](docs/endpoints/EquationApi.md#equations_id_put) | **PUT** /equations/{id} | Update an existing Equation
+*EquationApi* | [**equations_post**](docs/endpoints/EquationApi.md#equations_post) | **POST** /equations | Create one Equation
+*FundingInformationApi* | [**fundinginformations_get**](docs/endpoints/FundingInformationApi.md#fundinginformations_get) | **GET** /fundinginformations | List all instances of FundingInformation
+*FundingInformationApi* | [**fundinginformations_id_delete**](docs/endpoints/FundingInformationApi.md#fundinginformations_id_delete) | **DELETE** /fundinginformations/{id} | Delete an existing FundingInformation
+*FundingInformationApi* | [**fundinginformations_id_get**](docs/endpoints/FundingInformationApi.md#fundinginformations_id_get) | **GET** /fundinginformations/{id} | Get a single FundingInformation by its id
+*FundingInformationApi* | [**fundinginformations_id_put**](docs/endpoints/FundingInformationApi.md#fundinginformations_id_put) | **PUT** /fundinginformations/{id} | Update an existing FundingInformation
+*FundingInformationApi* | [**fundinginformations_post**](docs/endpoints/FundingInformationApi.md#fundinginformations_post) | **POST** /fundinginformations | Create one FundingInformation
+*GeoCoordinatesApi* | [**geocoordinatess_get**](docs/endpoints/GeoCoordinatesApi.md#geocoordinatess_get) | **GET** /geocoordinatess | List all instances of GeoCoordinates
+*GeoCoordinatesApi* | [**geocoordinatess_id_delete**](docs/endpoints/GeoCoordinatesApi.md#geocoordinatess_id_delete) | **DELETE** /geocoordinatess/{id} | Delete an existing GeoCoordinates
+*GeoCoordinatesApi* | [**geocoordinatess_id_get**](docs/endpoints/GeoCoordinatesApi.md#geocoordinatess_id_get) | **GET** /geocoordinatess/{id} | Get a single GeoCoordinates by its id
+*GeoCoordinatesApi* | [**geocoordinatess_id_put**](docs/endpoints/GeoCoordinatesApi.md#geocoordinatess_id_put) | **PUT** /geocoordinatess/{id} | Update an existing GeoCoordinates
+*GeoCoordinatesApi* | [**geocoordinatess_post**](docs/endpoints/GeoCoordinatesApi.md#geocoordinatess_post) | **POST** /geocoordinatess | Create one GeoCoordinates
+*GeoShapeApi* | [**geoshapes_get**](docs/endpoints/GeoShapeApi.md#geoshapes_get) | **GET** /geoshapes | List all instances of GeoShape
+*GeoShapeApi* | [**geoshapes_id_delete**](docs/endpoints/GeoShapeApi.md#geoshapes_id_delete) | **DELETE** /geoshapes/{id} | Delete an existing GeoShape
+*GeoShapeApi* | [**geoshapes_id_get**](docs/endpoints/GeoShapeApi.md#geoshapes_id_get) | **GET** /geoshapes/{id} | Get a single GeoShape by its id
+*GeoShapeApi* | [**geoshapes_id_put**](docs/endpoints/GeoShapeApi.md#geoshapes_id_put) | **PUT** /geoshapes/{id} | Update an existing GeoShape
+*GeoShapeApi* | [**geoshapes_post**](docs/endpoints/GeoShapeApi.md#geoshapes_post) | **POST** /geoshapes | Create one GeoShape
+*GridApi* | [**grids_get**](docs/endpoints/GridApi.md#grids_get) | **GET** /grids | List all instances of Grid
+*GridApi* | [**grids_id_delete**](docs/endpoints/GridApi.md#grids_id_delete) | **DELETE** /grids/{id} | Delete an existing Grid
+*GridApi* | [**grids_id_get**](docs/endpoints/GridApi.md#grids_id_get) | **GET** /grids/{id} | Get a single Grid by its id
+*GridApi* | [**grids_id_put**](docs/endpoints/GridApi.md#grids_id_put) | **PUT** /grids/{id} | Update an existing Grid
+*GridApi* | [**grids_post**](docs/endpoints/GridApi.md#grids_post) | **POST** /grids | Create one Grid
+*HybridModelApi* | [**hybridmodels_get**](docs/endpoints/HybridModelApi.md#hybridmodels_get) | **GET** /hybridmodels | List all instances of HybridModel
+*HybridModelApi* | [**hybridmodels_id_delete**](docs/endpoints/HybridModelApi.md#hybridmodels_id_delete) | **DELETE** /hybridmodels/{id} | Delete an existing HybridModel
+*HybridModelApi* | [**hybridmodels_id_get**](docs/endpoints/HybridModelApi.md#hybridmodels_id_get) | **GET** /hybridmodels/{id} | Get a single HybridModel by its id
+*HybridModelApi* | [**hybridmodels_id_put**](docs/endpoints/HybridModelApi.md#hybridmodels_id_put) | **PUT** /hybridmodels/{id} | Update an existing HybridModel
+*HybridModelApi* | [**hybridmodels_post**](docs/endpoints/HybridModelApi.md#hybridmodels_post) | **POST** /hybridmodels | Create one HybridModel
+*ImageApi* | [**images_get**](docs/endpoints/ImageApi.md#images_get) | **GET** /images | List all instances of Image
+*ImageApi* | [**images_id_delete**](docs/endpoints/ImageApi.md#images_id_delete) | **DELETE** /images/{id} | Delete an existing Image
+*ImageApi* | [**images_id_get**](docs/endpoints/ImageApi.md#images_id_get) | **GET** /images/{id} | Get a single Image by its id
+*ImageApi* | [**images_id_put**](docs/endpoints/ImageApi.md#images_id_put) | **PUT** /images/{id} | Update an existing Image
+*ImageApi* | [**images_post**](docs/endpoints/ImageApi.md#images_post) | **POST** /images | Create one Image
+*InterventionApi* | [**interventions_get**](docs/endpoints/InterventionApi.md#interventions_get) | **GET** /interventions | List all instances of Intervention
+*InterventionApi* | [**interventions_id_delete**](docs/endpoints/InterventionApi.md#interventions_id_delete) | **DELETE** /interventions/{id} | Delete an existing Intervention
+*InterventionApi* | [**interventions_id_get**](docs/endpoints/InterventionApi.md#interventions_id_get) | **GET** /interventions/{id} | Get a single Intervention by its id
+*InterventionApi* | [**interventions_id_put**](docs/endpoints/InterventionApi.md#interventions_id_put) | **PUT** /interventions/{id} | Update an existing Intervention
+*InterventionApi* | [**interventions_post**](docs/endpoints/InterventionApi.md#interventions_post) | **POST** /interventions | Create one Intervention
+*ModelApi* | [**custom_model_index_get**](docs/endpoints/ModelApi.md#custom_model_index_get) | **GET** /custom/model/index | Get a Model
+*ModelApi* | [**custom_model_intervention_get**](docs/endpoints/ModelApi.md#custom_model_intervention_get) | **GET** /custom/model/intervention | Get a Model
+*ModelApi* | [**custom_model_region_get**](docs/endpoints/ModelApi.md#custom_model_region_get) | **GET** /custom/model/region | Get a Model
+*ModelApi* | [**custom_models_standard_variable_get**](docs/endpoints/ModelApi.md#custom_models_standard_variable_get) | **GET** /custom/models/standard_variable | Get a list of models
+*ModelApi* | [**custom_models_variable_get**](docs/endpoints/ModelApi.md#custom_models_variable_get) | **GET** /custom/models/variable | Get a list of Model
+*ModelApi* | [**models_get**](docs/endpoints/ModelApi.md#models_get) | **GET** /models | List all instances of Model
+*ModelApi* | [**models_id_delete**](docs/endpoints/ModelApi.md#models_id_delete) | **DELETE** /models/{id} | Delete an existing Model
+*ModelApi* | [**models_id_get**](docs/endpoints/ModelApi.md#models_id_get) | **GET** /models/{id} | Get a single Model by its id
+*ModelApi* | [**models_id_put**](docs/endpoints/ModelApi.md#models_id_put) | **PUT** /models/{id} | Update an existing Model
+*ModelApi* | [**models_post**](docs/endpoints/ModelApi.md#models_post) | **POST** /models | Create one Model
+*ModelCategoryApi* | [**modelcategorys_get**](docs/endpoints/ModelCategoryApi.md#modelcategorys_get) | **GET** /modelcategorys | List all instances of ModelCategory
+*ModelCategoryApi* | [**modelcategorys_id_delete**](docs/endpoints/ModelCategoryApi.md#modelcategorys_id_delete) | **DELETE** /modelcategorys/{id} | Delete an existing ModelCategory
+*ModelCategoryApi* | [**modelcategorys_id_get**](docs/endpoints/ModelCategoryApi.md#modelcategorys_id_get) | **GET** /modelcategorys/{id} | Get a single ModelCategory by its id
+*ModelCategoryApi* | [**modelcategorys_id_put**](docs/endpoints/ModelCategoryApi.md#modelcategorys_id_put) | **PUT** /modelcategorys/{id} | Update an existing ModelCategory
+*ModelCategoryApi* | [**modelcategorys_post**](docs/endpoints/ModelCategoryApi.md#modelcategorys_post) | **POST** /modelcategorys | Create one ModelCategory
+*ModelConfigurationApi* | [**custom_modelconfigurations_id_get**](docs/endpoints/ModelConfigurationApi.md#custom_modelconfigurations_id_get) | **GET** /custom/modelconfigurations/{id} | Get a ModelConfiguration
+*ModelConfigurationApi* | [**modelconfigurations_get**](docs/endpoints/ModelConfigurationApi.md#modelconfigurations_get) | **GET** /modelconfigurations | List all instances of ModelConfiguration
+*ModelConfigurationApi* | [**modelconfigurations_id_delete**](docs/endpoints/ModelConfigurationApi.md#modelconfigurations_id_delete) | **DELETE** /modelconfigurations/{id} | Delete an existing ModelConfiguration
+*ModelConfigurationApi* | [**modelconfigurations_id_get**](docs/endpoints/ModelConfigurationApi.md#modelconfigurations_id_get) | **GET** /modelconfigurations/{id} | Get a single ModelConfiguration by its id
+*ModelConfigurationApi* | [**modelconfigurations_id_put**](docs/endpoints/ModelConfigurationApi.md#modelconfigurations_id_put) | **PUT** /modelconfigurations/{id} | Update an existing ModelConfiguration
+*ModelConfigurationApi* | [**modelconfigurations_post**](docs/endpoints/ModelConfigurationApi.md#modelconfigurations_post) | **POST** /modelconfigurations | Create one ModelConfiguration
+*ModelConfigurationSetupApi* | [**custom_modelconfigurationsetups_id_get**](docs/endpoints/ModelConfigurationSetupApi.md#custom_modelconfigurationsetups_id_get) | **GET** /custom/modelconfigurationsetups/{id} | Get a ModelConfigurationSetup
+*ModelConfigurationSetupApi* | [**custom_modelconfigurationsetups_variable_get**](docs/endpoints/ModelConfigurationSetupApi.md#custom_modelconfigurationsetups_variable_get) | **GET** /custom/modelconfigurationsetups/variable | Get a list  Model
+*ModelConfigurationSetupApi* | [**modelconfigurationsetups_get**](docs/endpoints/ModelConfigurationSetupApi.md#modelconfigurationsetups_get) | **GET** /modelconfigurationsetups | List all instances of ModelConfigurationSetup
+*ModelConfigurationSetupApi* | [**modelconfigurationsetups_id_delete**](docs/endpoints/ModelConfigurationSetupApi.md#modelconfigurationsetups_id_delete) | **DELETE** /modelconfigurationsetups/{id} | Delete an existing ModelConfigurationSetup
+*ModelConfigurationSetupApi* | [**modelconfigurationsetups_id_get**](docs/endpoints/ModelConfigurationSetupApi.md#modelconfigurationsetups_id_get) | **GET** /modelconfigurationsetups/{id} | Get a single ModelConfigurationSetup by its id
+*ModelConfigurationSetupApi* | [**modelconfigurationsetups_id_put**](docs/endpoints/ModelConfigurationSetupApi.md#modelconfigurationsetups_id_put) | **PUT** /modelconfigurationsetups/{id} | Update an existing ModelConfigurationSetup
+*ModelConfigurationSetupApi* | [**modelconfigurationsetups_post**](docs/endpoints/ModelConfigurationSetupApi.md#modelconfigurationsetups_post) | **POST** /modelconfigurationsetups | Create one ModelConfigurationSetup
+*NumericalIndexApi* | [**numericalindexs_get**](docs/endpoints/NumericalIndexApi.md#numericalindexs_get) | **GET** /numericalindexs | List all instances of NumericalIndex
+*NumericalIndexApi* | [**numericalindexs_id_delete**](docs/endpoints/NumericalIndexApi.md#numericalindexs_id_delete) | **DELETE** /numericalindexs/{id} | Delete an existing NumericalIndex
+*NumericalIndexApi* | [**numericalindexs_id_get**](docs/endpoints/NumericalIndexApi.md#numericalindexs_id_get) | **GET** /numericalindexs/{id} | Get a single NumericalIndex by its id
+*NumericalIndexApi* | [**numericalindexs_id_put**](docs/endpoints/NumericalIndexApi.md#numericalindexs_id_put) | **PUT** /numericalindexs/{id} | Update an existing NumericalIndex
+*NumericalIndexApi* | [**numericalindexs_post**](docs/endpoints/NumericalIndexApi.md#numericalindexs_post) | **POST** /numericalindexs | Create one NumericalIndex
+*OrganizationApi* | [**organizations_get**](docs/endpoints/OrganizationApi.md#organizations_get) | **GET** /organizations | List all instances of Organization
+*OrganizationApi* | [**organizations_id_delete**](docs/endpoints/OrganizationApi.md#organizations_id_delete) | **DELETE** /organizations/{id} | Delete an existing Organization
+*OrganizationApi* | [**organizations_id_get**](docs/endpoints/OrganizationApi.md#organizations_id_get) | **GET** /organizations/{id} | Get a single Organization by its id
+*OrganizationApi* | [**organizations_id_put**](docs/endpoints/OrganizationApi.md#organizations_id_put) | **PUT** /organizations/{id} | Update an existing Organization
+*OrganizationApi* | [**organizations_post**](docs/endpoints/OrganizationApi.md#organizations_post) | **POST** /organizations | Create one Organization
+*ParameterApi* | [**parameters_get**](docs/endpoints/ParameterApi.md#parameters_get) | **GET** /parameters | List all instances of Parameter
+*ParameterApi* | [**parameters_id_delete**](docs/endpoints/ParameterApi.md#parameters_id_delete) | **DELETE** /parameters/{id} | Delete an existing Parameter
+*ParameterApi* | [**parameters_id_get**](docs/endpoints/ParameterApi.md#parameters_id_get) | **GET** /parameters/{id} | Get a single Parameter by its id
+*ParameterApi* | [**parameters_id_put**](docs/endpoints/ParameterApi.md#parameters_id_put) | **PUT** /parameters/{id} | Update an existing Parameter
+*ParameterApi* | [**parameters_post**](docs/endpoints/ParameterApi.md#parameters_post) | **POST** /parameters | Create one Parameter
+*PersonApi* | [**persons_get**](docs/endpoints/PersonApi.md#persons_get) | **GET** /persons | List all instances of Person
+*PersonApi* | [**persons_id_delete**](docs/endpoints/PersonApi.md#persons_id_delete) | **DELETE** /persons/{id} | Delete an existing Person
+*PersonApi* | [**persons_id_get**](docs/endpoints/PersonApi.md#persons_id_get) | **GET** /persons/{id} | Get a single Person by its id
+*PersonApi* | [**persons_id_put**](docs/endpoints/PersonApi.md#persons_id_put) | **PUT** /persons/{id} | Update an existing Person
+*PersonApi* | [**persons_post**](docs/endpoints/PersonApi.md#persons_post) | **POST** /persons | Create one Person
+*PointBasedGridApi* | [**pointbasedgrids_get**](docs/endpoints/PointBasedGridApi.md#pointbasedgrids_get) | **GET** /pointbasedgrids | List all instances of PointBasedGrid
+*PointBasedGridApi* | [**pointbasedgrids_id_delete**](docs/endpoints/PointBasedGridApi.md#pointbasedgrids_id_delete) | **DELETE** /pointbasedgrids/{id} | Delete an existing PointBasedGrid
+*PointBasedGridApi* | [**pointbasedgrids_id_get**](docs/endpoints/PointBasedGridApi.md#pointbasedgrids_id_get) | **GET** /pointbasedgrids/{id} | Get a single PointBasedGrid by its id
+*PointBasedGridApi* | [**pointbasedgrids_id_put**](docs/endpoints/PointBasedGridApi.md#pointbasedgrids_id_put) | **PUT** /pointbasedgrids/{id} | Update an existing PointBasedGrid
+*PointBasedGridApi* | [**pointbasedgrids_post**](docs/endpoints/PointBasedGridApi.md#pointbasedgrids_post) | **POST** /pointbasedgrids | Create one PointBasedGrid
+*ProcessApi* | [**processs_get**](docs/endpoints/ProcessApi.md#processs_get) | **GET** /processs | List all instances of Process
+*ProcessApi* | [**processs_id_delete**](docs/endpoints/ProcessApi.md#processs_id_delete) | **DELETE** /processs/{id} | Delete an existing Process
+*ProcessApi* | [**processs_id_get**](docs/endpoints/ProcessApi.md#processs_id_get) | **GET** /processs/{id} | Get a single Process by its id
+*ProcessApi* | [**processs_id_put**](docs/endpoints/ProcessApi.md#processs_id_put) | **PUT** /processs/{id} | Update an existing Process
+*ProcessApi* | [**processs_post**](docs/endpoints/ProcessApi.md#processs_post) | **POST** /processs | Create one Process
+*RegionApi* | [**regions_get**](docs/endpoints/RegionApi.md#regions_get) | **GET** /regions | List all instances of Region
+*RegionApi* | [**regions_id_delete**](docs/endpoints/RegionApi.md#regions_id_delete) | **DELETE** /regions/{id} | Delete an existing Region
+*RegionApi* | [**regions_id_get**](docs/endpoints/RegionApi.md#regions_id_get) | **GET** /regions/{id} | Get a single Region by its id
+*RegionApi* | [**regions_id_put**](docs/endpoints/RegionApi.md#regions_id_put) | **PUT** /regions/{id} | Update an existing Region
+*RegionApi* | [**regions_post**](docs/endpoints/RegionApi.md#regions_post) | **POST** /regions | Create one Region
+*SampleCollectionApi* | [**samplecollections_get**](docs/endpoints/SampleCollectionApi.md#samplecollections_get) | **GET** /samplecollections | List all instances of SampleCollection
+*SampleCollectionApi* | [**samplecollections_id_delete**](docs/endpoints/SampleCollectionApi.md#samplecollections_id_delete) | **DELETE** /samplecollections/{id} | Delete an existing SampleCollection
+*SampleCollectionApi* | [**samplecollections_id_get**](docs/endpoints/SampleCollectionApi.md#samplecollections_id_get) | **GET** /samplecollections/{id} | Get a single SampleCollection by its id
+*SampleCollectionApi* | [**samplecollections_id_put**](docs/endpoints/SampleCollectionApi.md#samplecollections_id_put) | **PUT** /samplecollections/{id} | Update an existing SampleCollection
+*SampleCollectionApi* | [**samplecollections_post**](docs/endpoints/SampleCollectionApi.md#samplecollections_post) | **POST** /samplecollections | Create one SampleCollection
+*SampleExecutionApi* | [**sampleexecutions_get**](docs/endpoints/SampleExecutionApi.md#sampleexecutions_get) | **GET** /sampleexecutions | List all instances of SampleExecution
+*SampleExecutionApi* | [**sampleexecutions_id_delete**](docs/endpoints/SampleExecutionApi.md#sampleexecutions_id_delete) | **DELETE** /sampleexecutions/{id} | Delete an existing SampleExecution
+*SampleExecutionApi* | [**sampleexecutions_id_get**](docs/endpoints/SampleExecutionApi.md#sampleexecutions_id_get) | **GET** /sampleexecutions/{id} | Get a single SampleExecution by its id
+*SampleExecutionApi* | [**sampleexecutions_id_put**](docs/endpoints/SampleExecutionApi.md#sampleexecutions_id_put) | **PUT** /sampleexecutions/{id} | Update an existing SampleExecution
+*SampleExecutionApi* | [**sampleexecutions_post**](docs/endpoints/SampleExecutionApi.md#sampleexecutions_post) | **POST** /sampleexecutions | Create one SampleExecution
+*SampleResourceApi* | [**sampleresources_get**](docs/endpoints/SampleResourceApi.md#sampleresources_get) | **GET** /sampleresources | List all instances of SampleResource
+*SampleResourceApi* | [**sampleresources_id_delete**](docs/endpoints/SampleResourceApi.md#sampleresources_id_delete) | **DELETE** /sampleresources/{id} | Delete an existing SampleResource
+*SampleResourceApi* | [**sampleresources_id_get**](docs/endpoints/SampleResourceApi.md#sampleresources_id_get) | **GET** /sampleresources/{id} | Get a single SampleResource by its id
+*SampleResourceApi* | [**sampleresources_id_put**](docs/endpoints/SampleResourceApi.md#sampleresources_id_put) | **PUT** /sampleresources/{id} | Update an existing SampleResource
+*SampleResourceApi* | [**sampleresources_post**](docs/endpoints/SampleResourceApi.md#sampleresources_post) | **POST** /sampleresources | Create one SampleResource
+*SoftwareApi* | [**softwares_get**](docs/endpoints/SoftwareApi.md#softwares_get) | **GET** /softwares | List all instances of Software
+*SoftwareApi* | [**softwares_id_delete**](docs/endpoints/SoftwareApi.md#softwares_id_delete) | **DELETE** /softwares/{id} | Delete an existing Software
+*SoftwareApi* | [**softwares_id_get**](docs/endpoints/SoftwareApi.md#softwares_id_get) | **GET** /softwares/{id} | Get a single Software by its id
+*SoftwareApi* | [**softwares_id_put**](docs/endpoints/SoftwareApi.md#softwares_id_put) | **PUT** /softwares/{id} | Update an existing Software
+*SoftwareApi* | [**softwares_post**](docs/endpoints/SoftwareApi.md#softwares_post) | **POST** /softwares | Create one Software
+*SoftwareConfigurationApi* | [**softwareconfigurations_get**](docs/endpoints/SoftwareConfigurationApi.md#softwareconfigurations_get) | **GET** /softwareconfigurations | List all instances of SoftwareConfiguration
+*SoftwareConfigurationApi* | [**softwareconfigurations_id_delete**](docs/endpoints/SoftwareConfigurationApi.md#softwareconfigurations_id_delete) | **DELETE** /softwareconfigurations/{id} | Delete an existing SoftwareConfiguration
+*SoftwareConfigurationApi* | [**softwareconfigurations_id_get**](docs/endpoints/SoftwareConfigurationApi.md#softwareconfigurations_id_get) | **GET** /softwareconfigurations/{id} | Get a single SoftwareConfiguration by its id
+*SoftwareConfigurationApi* | [**softwareconfigurations_id_put**](docs/endpoints/SoftwareConfigurationApi.md#softwareconfigurations_id_put) | **PUT** /softwareconfigurations/{id} | Update an existing SoftwareConfiguration
+*SoftwareConfigurationApi* | [**softwareconfigurations_post**](docs/endpoints/SoftwareConfigurationApi.md#softwareconfigurations_post) | **POST** /softwareconfigurations | Create one SoftwareConfiguration
+*SoftwareImageApi* | [**softwareimages_get**](docs/endpoints/SoftwareImageApi.md#softwareimages_get) | **GET** /softwareimages | List all instances of SoftwareImage
+*SoftwareImageApi* | [**softwareimages_id_delete**](docs/endpoints/SoftwareImageApi.md#softwareimages_id_delete) | **DELETE** /softwareimages/{id} | Delete an existing SoftwareImage
+*SoftwareImageApi* | [**softwareimages_id_get**](docs/endpoints/SoftwareImageApi.md#softwareimages_id_get) | **GET** /softwareimages/{id} | Get a single SoftwareImage by its id
+*SoftwareImageApi* | [**softwareimages_id_put**](docs/endpoints/SoftwareImageApi.md#softwareimages_id_put) | **PUT** /softwareimages/{id} | Update an existing SoftwareImage
+*SoftwareImageApi* | [**softwareimages_post**](docs/endpoints/SoftwareImageApi.md#softwareimages_post) | **POST** /softwareimages | Create one SoftwareImage
+*SoftwareVersionApi* | [**softwareversions_get**](docs/endpoints/SoftwareVersionApi.md#softwareversions_get) | **GET** /softwareversions | List all instances of SoftwareVersion
+*SoftwareVersionApi* | [**softwareversions_id_delete**](docs/endpoints/SoftwareVersionApi.md#softwareversions_id_delete) | **DELETE** /softwareversions/{id} | Delete an existing SoftwareVersion
+*SoftwareVersionApi* | [**softwareversions_id_get**](docs/endpoints/SoftwareVersionApi.md#softwareversions_id_get) | **GET** /softwareversions/{id} | Get a single SoftwareVersion by its id
+*SoftwareVersionApi* | [**softwareversions_id_put**](docs/endpoints/SoftwareVersionApi.md#softwareversions_id_put) | **PUT** /softwareversions/{id} | Update an existing SoftwareVersion
+*SoftwareVersionApi* | [**softwareversions_post**](docs/endpoints/SoftwareVersionApi.md#softwareversions_post) | **POST** /softwareversions | Create one SoftwareVersion
+*SourceCodeApi* | [**sourcecodes_get**](docs/endpoints/SourceCodeApi.md#sourcecodes_get) | **GET** /sourcecodes | List all instances of SourceCode
+*SourceCodeApi* | [**sourcecodes_id_delete**](docs/endpoints/SourceCodeApi.md#sourcecodes_id_delete) | **DELETE** /sourcecodes/{id} | Delete an existing SourceCode
+*SourceCodeApi* | [**sourcecodes_id_get**](docs/endpoints/SourceCodeApi.md#sourcecodes_id_get) | **GET** /sourcecodes/{id} | Get a single SourceCode by its id
+*SourceCodeApi* | [**sourcecodes_id_put**](docs/endpoints/SourceCodeApi.md#sourcecodes_id_put) | **PUT** /sourcecodes/{id} | Update an existing SourceCode
+*SourceCodeApi* | [**sourcecodes_post**](docs/endpoints/SourceCodeApi.md#sourcecodes_post) | **POST** /sourcecodes | Create one SourceCode
+*SpatialResolutionApi* | [**spatialresolutions_get**](docs/endpoints/SpatialResolutionApi.md#spatialresolutions_get) | **GET** /spatialresolutions | List all instances of SpatialResolution
+*SpatialResolutionApi* | [**spatialresolutions_id_delete**](docs/endpoints/SpatialResolutionApi.md#spatialresolutions_id_delete) | **DELETE** /spatialresolutions/{id} | Delete an existing SpatialResolution
+*SpatialResolutionApi* | [**spatialresolutions_id_get**](docs/endpoints/SpatialResolutionApi.md#spatialresolutions_id_get) | **GET** /spatialresolutions/{id} | Get a single SpatialResolution by its id
+*SpatialResolutionApi* | [**spatialresolutions_id_put**](docs/endpoints/SpatialResolutionApi.md#spatialresolutions_id_put) | **PUT** /spatialresolutions/{id} | Update an existing SpatialResolution
+*SpatialResolutionApi* | [**spatialresolutions_post**](docs/endpoints/SpatialResolutionApi.md#spatialresolutions_post) | **POST** /spatialresolutions | Create one SpatialResolution
+*SpatiallyDistributedGridApi* | [**spatiallydistributedgrids_get**](docs/endpoints/SpatiallyDistributedGridApi.md#spatiallydistributedgrids_get) | **GET** /spatiallydistributedgrids | List all instances of SpatiallyDistributedGrid
+*SpatiallyDistributedGridApi* | [**spatiallydistributedgrids_id_delete**](docs/endpoints/SpatiallyDistributedGridApi.md#spatiallydistributedgrids_id_delete) | **DELETE** /spatiallydistributedgrids/{id} | Delete an existing SpatiallyDistributedGrid
+*SpatiallyDistributedGridApi* | [**spatiallydistributedgrids_id_get**](docs/endpoints/SpatiallyDistributedGridApi.md#spatiallydistributedgrids_id_get) | **GET** /spatiallydistributedgrids/{id} | Get a single SpatiallyDistributedGrid by its id
+*SpatiallyDistributedGridApi* | [**spatiallydistributedgrids_id_put**](docs/endpoints/SpatiallyDistributedGridApi.md#spatiallydistributedgrids_id_put) | **PUT** /spatiallydistributedgrids/{id} | Update an existing SpatiallyDistributedGrid
+*SpatiallyDistributedGridApi* | [**spatiallydistributedgrids_post**](docs/endpoints/SpatiallyDistributedGridApi.md#spatiallydistributedgrids_post) | **POST** /spatiallydistributedgrids | Create one SpatiallyDistributedGrid
+*StandardVariableApi* | [**standardvariables_get**](docs/endpoints/StandardVariableApi.md#standardvariables_get) | **GET** /standardvariables | List all instances of StandardVariable
+*StandardVariableApi* | [**standardvariables_id_delete**](docs/endpoints/StandardVariableApi.md#standardvariables_id_delete) | **DELETE** /standardvariables/{id} | Delete an existing StandardVariable
+*StandardVariableApi* | [**standardvariables_id_get**](docs/endpoints/StandardVariableApi.md#standardvariables_id_get) | **GET** /standardvariables/{id} | Get a single StandardVariable by its id
+*StandardVariableApi* | [**standardvariables_id_put**](docs/endpoints/StandardVariableApi.md#standardvariables_id_put) | **PUT** /standardvariables/{id} | Update an existing StandardVariable
+*StandardVariableApi* | [**standardvariables_post**](docs/endpoints/StandardVariableApi.md#standardvariables_post) | **POST** /standardvariables | Create one StandardVariable
+*TheoryGuidedModelApi* | [**theory_guidedmodels_get**](docs/endpoints/TheoryGuidedModelApi.md#theory_guidedmodels_get) | **GET** /theory-guidedmodels | List all instances of Theory-GuidedModel
+*TheoryGuidedModelApi* | [**theory_guidedmodels_id_delete**](docs/endpoints/TheoryGuidedModelApi.md#theory_guidedmodels_id_delete) | **DELETE** /theory-guidedmodels/{id} | Delete an existing Theory-GuidedModel
+*TheoryGuidedModelApi* | [**theory_guidedmodels_id_get**](docs/endpoints/TheoryGuidedModelApi.md#theory_guidedmodels_id_get) | **GET** /theory-guidedmodels/{id} | Get a single Theory-GuidedModel by its id
+*TheoryGuidedModelApi* | [**theory_guidedmodels_id_put**](docs/endpoints/TheoryGuidedModelApi.md#theory_guidedmodels_id_put) | **PUT** /theory-guidedmodels/{id} | Update an existing Theory-GuidedModel
+*TheoryGuidedModelApi* | [**theory_guidedmodels_post**](docs/endpoints/TheoryGuidedModelApi.md#theory_guidedmodels_post) | **POST** /theory-guidedmodels | Create one Theory-GuidedModel
+*TimeIntervalApi* | [**timeintervals_get**](docs/endpoints/TimeIntervalApi.md#timeintervals_get) | **GET** /timeintervals | List all instances of TimeInterval
+*TimeIntervalApi* | [**timeintervals_id_delete**](docs/endpoints/TimeIntervalApi.md#timeintervals_id_delete) | **DELETE** /timeintervals/{id} | Delete an existing TimeInterval
+*TimeIntervalApi* | [**timeintervals_id_get**](docs/endpoints/TimeIntervalApi.md#timeintervals_id_get) | **GET** /timeintervals/{id} | Get a single TimeInterval by its id
+*TimeIntervalApi* | [**timeintervals_id_put**](docs/endpoints/TimeIntervalApi.md#timeintervals_id_put) | **PUT** /timeintervals/{id} | Update an existing TimeInterval
+*TimeIntervalApi* | [**timeintervals_post**](docs/endpoints/TimeIntervalApi.md#timeintervals_post) | **POST** /timeintervals | Create one TimeInterval
+*UnitApi* | [**units_get**](docs/endpoints/UnitApi.md#units_get) | **GET** /units | List all instances of Unit
+*UnitApi* | [**units_id_delete**](docs/endpoints/UnitApi.md#units_id_delete) | **DELETE** /units/{id} | Delete an existing Unit
+*UnitApi* | [**units_id_get**](docs/endpoints/UnitApi.md#units_id_get) | **GET** /units/{id} | Get a single Unit by its id
+*UnitApi* | [**units_id_put**](docs/endpoints/UnitApi.md#units_id_put) | **PUT** /units/{id} | Update an existing Unit
+*UnitApi* | [**units_post**](docs/endpoints/UnitApi.md#units_post) | **POST** /units | Create one Unit
+*VariableApi* | [**variables_get**](docs/endpoints/VariableApi.md#variables_get) | **GET** /variables | List all instances of Variable
+*VariableApi* | [**variables_id_delete**](docs/endpoints/VariableApi.md#variables_id_delete) | **DELETE** /variables/{id} | Delete an existing Variable
+*VariableApi* | [**variables_id_get**](docs/endpoints/VariableApi.md#variables_id_get) | **GET** /variables/{id} | Get a single Variable by its id
+*VariableApi* | [**variables_id_put**](docs/endpoints/VariableApi.md#variables_id_put) | **PUT** /variables/{id} | Update an existing Variable
+*VariableApi* | [**variables_post**](docs/endpoints/VariableApi.md#variables_post) | **POST** /variables | Create one Variable
+*VariablePresentationApi* | [**variablepresentations_get**](docs/endpoints/VariablePresentationApi.md#variablepresentations_get) | **GET** /variablepresentations | List all instances of VariablePresentation
+*VariablePresentationApi* | [**variablepresentations_id_delete**](docs/endpoints/VariablePresentationApi.md#variablepresentations_id_delete) | **DELETE** /variablepresentations/{id} | Delete an existing VariablePresentation
+*VariablePresentationApi* | [**variablepresentations_id_get**](docs/endpoints/VariablePresentationApi.md#variablepresentations_id_get) | **GET** /variablepresentations/{id} | Get a single VariablePresentation by its id
+*VariablePresentationApi* | [**variablepresentations_id_put**](docs/endpoints/VariablePresentationApi.md#variablepresentations_id_put) | **PUT** /variablepresentations/{id} | Update an existing VariablePresentation
+*VariablePresentationApi* | [**variablepresentations_post**](docs/endpoints/VariablePresentationApi.md#variablepresentations_post) | **POST** /variablepresentations | Create one VariablePresentation
+*VisualizationApi* | [**visualizations_get**](docs/endpoints/VisualizationApi.md#visualizations_get) | **GET** /visualizations | List all instances of Visualization
+*VisualizationApi* | [**visualizations_id_delete**](docs/endpoints/VisualizationApi.md#visualizations_id_delete) | **DELETE** /visualizations/{id} | Delete an existing Visualization
+*VisualizationApi* | [**visualizations_id_get**](docs/endpoints/VisualizationApi.md#visualizations_id_get) | **GET** /visualizations/{id} | Get a single Visualization by its id
+*VisualizationApi* | [**visualizations_id_put**](docs/endpoints/VisualizationApi.md#visualizations_id_put) | **PUT** /visualizations/{id} | Update an existing Visualization
+*VisualizationApi* | [**visualizations_post**](docs/endpoints/VisualizationApi.md#visualizations_post) | **POST** /visualizations | Create one Visualization
+*DefaultApi* | [**user_login_post**](docs/endpoints/DefaultApi.md#user_login_post) | **POST** /user/login | 
+
+
+## Documentation For Models
+
+ - [CatalogIdentifier](docs/endpoints/CatalogIdentifier.md)
+ - [CausalDiagram](docs/endpoints/CausalDiagram.md)
+ - [ConfigurationSetup](docs/endpoints/ConfigurationSetup.md)
+ - [Constraint](docs/endpoints/Constraint.md)
+ - [CoupledModel](docs/endpoints/CoupledModel.md)
+ - [DataTransformation](docs/endpoints/DataTransformation.md)
+ - [DataTransformationSetup](docs/endpoints/DataTransformationSetup.md)
+ - [DatasetSpecification](docs/endpoints/DatasetSpecification.md)
+ - [EmpiricalModel](docs/endpoints/EmpiricalModel.md)
+ - [Emulator](docs/endpoints/Emulator.md)
+ - [Equation](docs/endpoints/Equation.md)
+ - [FundingInformation](docs/endpoints/FundingInformation.md)
+ - [GeoCoordinates](docs/endpoints/GeoCoordinates.md)
+ - [GeoShape](docs/endpoints/GeoShape.md)
+ - [Grid](docs/endpoints/Grid.md)
+ - [HybridModel](docs/endpoints/HybridModel.md)
+ - [Image](docs/endpoints/Image.md)
+ - [Intervention](docs/endpoints/Intervention.md)
+ - [Model](docs/endpoints/Model.md)
+ - [ModelCategory](docs/endpoints/ModelCategory.md)
+ - [ModelConfiguration](docs/endpoints/ModelConfiguration.md)
+ - [ModelConfigurationSetup](docs/endpoints/ModelConfigurationSetup.md)
+ - [NumericalIndex](docs/endpoints/NumericalIndex.md)
+ - [Organization](docs/endpoints/Organization.md)
+ - [Parameter](docs/endpoints/Parameter.md)
+ - [Person](docs/endpoints/Person.md)
+ - [PointBasedGrid](docs/endpoints/PointBasedGrid.md)
+ - [Process](docs/endpoints/Process.md)
+ - [Region](docs/endpoints/Region.md)
+ - [SampleCollection](docs/endpoints/SampleCollection.md)
+ - [SampleExecution](docs/endpoints/SampleExecution.md)
+ - [SampleResource](docs/endpoints/SampleResource.md)
+ - [Software](docs/endpoints/Software.md)
+ - [SoftwareConfiguration](docs/endpoints/SoftwareConfiguration.md)
+ - [SoftwareImage](docs/endpoints/SoftwareImage.md)
+ - [SoftwareVersion](docs/endpoints/SoftwareVersion.md)
+ - [SourceCode](docs/endpoints/SourceCode.md)
+ - [SpatialResolution](docs/endpoints/SpatialResolution.md)
+ - [SpatiallyDistributedGrid](docs/endpoints/SpatiallyDistributedGrid.md)
+ - [StandardVariable](docs/endpoints/StandardVariable.md)
+ - [TheoryGuidedModel](docs/endpoints/TheoryGuidedModel.md)
+ - [TimeInterval](docs/endpoints/TimeInterval.md)
+ - [Unit](docs/endpoints/Unit.md)
+ - [User](docs/endpoints/User.md)
+ - [Variable](docs/endpoints/Variable.md)
+ - [VariablePresentation](docs/endpoints/VariablePresentation.md)
+ - [Visualization](docs/endpoints/Visualization.md)
+
+
+## Documentation For Authorization
+
+
+## BearerAuth
+
+- **Type**: Bearer authentication (JWT)
+
+
+## Author
+
+
+
+
+
diff --git a/src/somef/test/test_data/test_logo.txt b/src/somef/test/test_data/test_logo.txt
index 04422a97..f8e5b260 100644
--- a/src/somef/test/test_data/test_logo.txt
+++ b/src/somef/test/test_data/test_logo.txt
@@ -1,4 +1,4 @@
-![Logo](https://github.com/oeg-upm/Chowlk/blob/webservice/static/resources/logo.png)
+![Logo](https://github.com/oeg-upm/Chowlk/blob/webservice/resources/logo.png)
 
 # Chowlk Converter
 Tool to transform ontology conceptualizations made with diagrams.net into OWL code.
diff --git a/src/somef/test/test_data/test_pypi.md b/src/somef/test/test_data/test_pypi.md
new file mode 100644
index 00000000..a6113570
--- /dev/null
+++ b/src/somef/test/test_data/test_pypi.md
@@ -0,0 +1,26 @@
+# inspect4py
+[![PyPI](https://badge.fury.io/py/inspect4py.svg)](https://badge.fury.io/py/inspect4py) [![DOI](https://zenodo.org/badge/349160905.svg)](https://zenodo.org/badge/latestdoi/349160905) [![Project Status: Active – The project has reached a stable, usable state and is being actively developed.](https://www.repostatus.org/badges/latest/active.svg)](https://www.repostatus.org/#active)
+
+<img src="docs/images/logo.png" alt="logo" width="200"/>
+
+Library to allow users inspect a software project folder (i.e., a directory and its subdirectories) and extract all the most relevant information, such as class, method and parameter documentation, classes (and their methods), functions, etc.
+
+## Features:
+
+Given a folder with code, `inspect4py` will:
+
+- Extract all imported modules and how each module is imported as (i.e., whether they are internal or external).
+- Extract all functions in the code, including their documentation, parameters, accepted values, and call list.
+- Extract all classes in the code, with all their methods and respective documentation
+- Extract the control flow of each file.
+- Extract the hierarchy of directories and files.
+- Extract the requirements used in the software project.
+- Classify which files are tests
+- Classify the main type of software project (script, package, library or service). Only one type is returned as main type (e.g., if a library has the option to be deployed as a service, `inspect4py` will return `Library` as its main type)
+- Return a ranking of the different ways in which a a software component can be run, ordered by relevance.
+
+
+All metadata is extracted as a JSON file.
+
+
+Inspect4py currently works **only for Python 3 projects**.
diff --git a/src/somef/test/test_data/test_wiki.md b/src/somef/test/test_data/test_wiki.md
new file mode 100644
index 00000000..963be8fc
--- /dev/null
+++ b/src/somef/test/test_data/test_wiki.md
@@ -0,0 +1,46 @@
+ [![Generic badge](https://img.shields.io/badge/Status-Developing-yellow)](https://shields.io/)
+ [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://github.com/oeg-upm/Mapeathor/blob/master/LICENSE)
+ [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.5973906.svg)](https://doi.org/10.5281/zenodo.5973906)
+ [![version](https://img.shields.io/badge/python-3.6+-blue.svg)](https://www.python.org/downloads/release/python-360/)
+ [![Latest PyPI version](https://img.shields.io/pypi/v/mapeathor?style=flat)](https://pypi.python.org/pypi/mapeathor)
+
+<p align="center">
+ <img src="https://raw.githubusercontent.com/oeg-upm/Mapeathor/master/imgs/Square_logo_mapeathor.png" alt="workflow" width="150"/>
+</p>
+
+
+# Mapeathor
+#### Mapeathor translates your mapping rules specified in spreadsheets to a mapping language.  
+
+[Mapeathor](https://morph.oeg.fi.upm.es/tool/mapeathor) is a simple spreadsheet parser able to generate mapping rules in three mapping languages: R2RML, RML (with extension to functions from FnO) and YARRRML. It takes the mapping rules expressed in a spreadsheet and transforms them into the desired language. The spreadsheet template is designed to facilitate the mapping rules' writting, with the aim of being language independent, and thus, lowering the barrier of generating mappings for non-expert users.
+
+<p align="center">
+ <img src="https://raw.githubusercontent.com/oeg-upm/Mapeathor/master/imgs/workflow.png" alt="workflow" width="600"/>
+</p>
+
+## Example
+A more detailed explanation is provided in the [wiki](https://github.com/oeg-upm/Mapeathor/wiki).
+
+### First Step: Fill the xlsx template with the transformation rules
+The template has five mandatory sheets, *Prefixes, Source, Subject PredicateObjectMap* and *Functions*. The last one can be left blank in case there are no functions. The spreadsheet can be in XLSX format or a Google Spreadsheet. Here is an example of the structure of the spreadsheet.
+
+<p align="center">
+ <img src="https://raw.githubusercontent.com/oeg-upm/Mapeathor/master/imgs/sheets.png" alt="sheets" width="500"/>
+</p>
+
+### Second Step: Choose the output language
+One of three options can be chosen: R2RML, RML or YARRRML.
+
+### Third Step: Run it!
+The easiest way of running Mapeathor is using the [web service](https://morph.oeg.fi.upm.es/demo/mapeathor) and the [Swagger](https://morph.oeg.fi.upm.es/tool/mapeathor/swagger/) instance. For CLI lovers, the service is available as a [PyPi package](https://pypi.org/project/mapeathor/) and Docker image. The instructions of the latest can be found in the [wiki](https://github.com/oeg-upm/Mapeathor/wiki).
+
+## Publications
+Iglesias-Molina, A., Pozo-Gilo, L., Dona, D., Ruckhaus, E., Chaves-Fraga, D., & Corcho, O. (2020, January). *Mapeathor: Simplifying the Specification of Declarative Rules for Knowledge Graph Construction. In ISWC (Demos/Industry).* [Online version](http://ceur-ws.org/Vol-2721/paper488.pdf)
+
+Iglesias-Molina, A., Chaves-Fraga, D., Priyatna, F., & Corcho, O. (2019). Towards the Definition of a Language-Independent Mapping Template for Knowledge Graph Creation. *In Proceedings of the Third International Workshop on Capturing Scientific Knowledge co-located with the 10th International Conference on Knowledge Capture (K-CAP 2019)* (pp. 33-36). [Online version](https://sciknow.github.io/sciknow2019/papers/SciKnow_2019_paper_4.pdf)
+
+## Authors and contact
+- [Ana Iglesias-Molina](https://github.com/anaigmo) - [ana.iglesiasm@upm.es](mailto:ana.iglesiasm@upm.es)
+- [Luis Pozo](https://github.com/w0xter)
+- [Daniel Doña](https://github.com/daniel-dona)
+
diff --git a/src/somef/test/test_header_analysis.py b/src/somef/test/test_header_analysis.py
index ffbb7cb4..6ba09825 100644
--- a/src/somef/test/test_header_analysis.py
+++ b/src/somef/test/test_header_analysis.py
@@ -4,7 +4,9 @@
 
 from pathlib import Path
 
-from ..header_analysis import extract_header_content, extract_categories_using_headers, extract_bash_code
+from ..header_analysis import extract_header_content, extract_categories, extract_bash_code
+from ..process_results import Result
+from ..utils import constants
 
 # Test data for tests
 test_data_path = str(Path(__file__).parent / "test_data") + os.path.sep
@@ -33,7 +35,7 @@ def test_extract_header_content_underline(self):
         """Parser checks in case header are underlined (e.g., '___')"""
         with open(test_data_path + "test_extract_header_content_underline.txt", "r") as data_file:
             text = data_file.read()
-            result, non_header_content  = extract_header_content(text)
+            result, non_header_content = extract_header_content(text)
             print(result)
             assert len(result.index) == 3
 
@@ -41,9 +43,9 @@ def test_extract_categories_using_headers(self):
         """More markdown checks to see if all categories are recognized."""
         with open(test_data_path + "widoco_readme.md", "r") as data_file:
             file_text = data_file.read()
-            json, results = extract_categories_using_headers(file_text)
+            result, list = extract_categories(file_text, Result())
             # At least 5 categories are extracted from the header analysis
-            assert len(json) == 5
+            assert len(result.results) >= 5
 
     def test_extract_bash_code(self):
         """Test to check if bash code can be detected in readme files"""
@@ -69,7 +71,7 @@ def test_issue_237(self):
         """Test to check excerpt division and classification"""
         with open(test_data_path + "README-manim.md", "r") as data_file:
             file_text = data_file.read()
-            json, results = extract_categories_using_headers(file_text)
+            json_results, results = extract_categories(file_text, Result())
             element = results[0]
             split = element.split("\n")
             assert len(split) > 1
@@ -79,19 +81,28 @@ def test_issue_313(self):
         """Test to see if the original title of a section is returned"""
         with open(test_data_path + "README-manim.md", "r") as data_file:
             file_text = data_file.read()
-            json, results = extract_categories_using_headers(file_text)
-            element = json.get('documentation')
-            elem = element[0]
-            title = elem.get('originalHeader')
+            json, results = extract_categories(file_text, Result())
+            element = json.results[constants.CAT_DOCUMENTATION]
+            elem = element[0][constants.PROP_RESULT]
+            title = elem.get(constants.PROP_ORIGINAL_HEADER)
             assert title == 'Documentation'
 
     def test_issue_425(self):
         """Checks that the confidence value of fields extracted using header extraction technique is 1.0"""
         with open(test_data_path + "README-mapshaper.md", "r") as data_file:
             file_text = data_file.read()
-            json, results = extract_categories_using_headers(file_text)
-            element = json.get('description')
-            elem = element[0]
-            confidence = elem.get('confidence')
-            print(confidence)
-            assert confidence == [1.0]
\ No newline at end of file
+            json, results = extract_categories(file_text, Result())
+            element = json.results[constants.CAT_DESCRIPTION]
+            confidence = element[0][constants.PROP_CONFIDENCE]
+            assert confidence == 1
+
+    def test_issue_465(self):
+        """
+        Test targeted towards creating the right excerpts when breaking a problematic file. Requirements should
+        only return a single line.
+        """
+        with open(test_data_path + "test_465.md", "r") as data_file:
+            file_text = data_file.read()
+            json_test, results = extract_categories(file_text, Result())
+            reqs = json_test.results[constants.CAT_REQUIREMENTS][0][constants.PROP_RESULT][constants.PROP_VALUE]
+            assert reqs.replace('\n', '') == "Python 2.7 and 3.4+"
diff --git a/src/somef/test/test_process_repository.py b/src/somef/test/test_process_repository.py
index 1e404702..f526790e 100644
--- a/src/somef/test/test_process_repository.py
+++ b/src/somef/test/test_process_repository.py
@@ -5,6 +5,7 @@
 
 from .. import process_repository, process_files, somef_cli
 from ..utils import constants
+from ..process_results import Result
 
 test_data_repositories = str(Path(__file__).parent / "test_data" / "repositories") + os.path.sep
 
@@ -13,63 +14,84 @@ class TestProcessRepository(unittest.TestCase):
 
     def test_issue_171(self):
         """Test designed to check if contributors are detected"""
-        text, github_data = process_files.process_repository_files(test_data_repositories + "rdflib-6.0.2", {},
+        res = Result()
+        text, res = process_files.process_repository_files(test_data_repositories + "rdflib-6.0.2", res,
                                                                    constants.RepositoryType.LOCAL)
-        assert len(github_data['contributors']) > 0
+        assert len(res.results[constants.CAT_CONTRIBUTORS]) > 0
 
     def test_issue_209(self):
         """Test designed to check if sh scripts are detected"""
-        text, github_data = process_files.process_repository_files(test_data_repositories + "rdflib-6.0.2", {},
+        github_data = Result()
+        text, github_data = process_files.process_repository_files(test_data_repositories + "rdflib-6.0.2", github_data,
                                                                    constants.RepositoryType.LOCAL)
-        assert len(github_data['hasScriptFile']) > 0
+        assert len(github_data.results[constants.CAT_HAS_SCRIPT_FILE]) > 0
 
     def test_issue_211(self):
         """Test designed to check if contributor guidelines are detected"""
-        text, github_data = process_files.process_repository_files(test_data_repositories + "probot-12.1.1", {},
+        github_data = Result()
+        text, github_data = process_files.process_repository_files(test_data_repositories + "probot-12.1.1", github_data,
                                                                    constants.RepositoryType.LOCAL)
-        assert len(github_data['contributingGuidelines']) > 0 and (
-                'license' in github_data.keys() or 'licenseText' in github_data.keys())
+        assert len(github_data.results[constants.CAT_CONTRIBUTING_GUIDELINES]) > 0 and (
+                'license' in github_data.results.keys())
+
+    def test_coc(self):
+        """Test designed to check if code of conduct is detected"""
+        github_data = Result()
+        text, github_data = process_files.process_repository_files(test_data_repositories + "wav2letter", github_data,
+                                                                   constants.RepositoryType.LOCAL)
+        assert len(github_data.results[constants.CAT_COC]) > 0
+
+    def test_install(self):
+        """Test designed to check if installation instructions are detected"""
+        github_data = Result()
+        text, github_data = process_files.process_repository_files(test_data_repositories + "wav2letter", github_data,
+                                                                   constants.RepositoryType.LOCAL)
+        assert len(github_data.results[constants.CAT_INSTALLATION]) > 0
 
     def test_issue_218(self):
         """Test designed to check if citation files (.cff) are detected"""
-        text, github_data = process_files.process_repository_files(test_data_repositories + "captum", {},
+        github_data = Result()
+        text, github_data = process_files.process_repository_files(test_data_repositories + "captum", github_data,
                                                                    constants.RepositoryType.LOCAL)
-        assert len(github_data['citation']) > 0
+        assert len(github_data.results[constants.CAT_CITATION]) > 0
 
     def test_issue_285(self):
         """Test designed to test repositories with no license (somef should not break)"""
-
+        github_data = Result()
         text, github_data = process_files.process_repository_files(test_data_repositories +
-                                                                   "repos-oeg/cogito-kgg-module", {},
+                                                                   "repos-oeg/cogito-kgg-module", github_data,
                                                                    constants.RepositoryType.LOCAL)
-        assert 'license' not in github_data
+        assert 'license' not in github_data.results
 
     def test_issue_361(self):
         """Test designed to check if multiple readme combinations are detected"""
+        github_data = Result()
         text, github_data = process_files.process_repository_files(test_data_repositories +
-                                                                   "rdflib-6.0.2", {},
+                                                                   "rdflib-6.0.2", github_data,
                                                                    constants.RepositoryType.LOCAL)
         assert len(text) > 0
 
     def test_issue_389(self):
         """Test designed to check if Dockerfiles are detected"""
+        github_data = Result()
         text, github_data = process_files.process_repository_files(test_data_repositories +
-                                                                   "wot-hive", {},
+                                                                   "wot-hive", github_data,
                                                                    constants.RepositoryType.LOCAL)
-        assert len(github_data['hasBuildFile']) > 0
+        assert len(github_data.results[constants.CAT_HAS_BUILD_FILE]) > 0
 
     def test_issue_no_readme(self):
         """Test designed to check if repositories with no readme are detected"""
-        github_data, owner, repo, def_br = process_repository.load_online_repository_metadata(
+        github_data, owner, repo, def_br = process_repository.load_online_repository_metadata(Result(),
             "https://github.com/oeg-upm/OpenRefineExtension_Transformation")
-        assert 'codeRepository' in github_data
+        assert constants.CAT_CODE_REPOSITORY in github_data.results.keys()
 
     def test_issue_268(self):
         """Test designed to check if license is properly captured"""
+        res = Result()
         text, github_data = process_files.process_repository_files(test_data_repositories +
-                                                                   "probot-12.1.1", {},
+                                                                   "probot-12.1.1", res,
                                                                    constants.RepositoryType.LOCAL)
-        assert len(github_data['licenseText']) > 0
+        assert len(github_data.results[constants.CAT_LICENSE]) > 0
 
     # Commenting this issue: this repo does no longer have an ACK file
     # def test_issue_210(self):
@@ -92,28 +114,31 @@ def test_issue_268(self):
 
     def test_issue_286(self):
         """Test designed to check if gitlab repositories are properly parsed"""
-        gitlab_data, owner, name, def_br = process_repository.load_gitlab_repository_metadata(
+        gitlab_data, owner, name, def_br = process_repository.load_gitlab_repository_metadata(Result(),
             "https://gitlab.com/gitlab-org/ci-sample-projects/platform-team")
-        assert len(gitlab_data['downloadUrl']) > 0
+        # print(gitlab_data.results)
+        assert len(gitlab_data.results[constants.CAT_DOWNLOAD_URL]) > 0
 
-    def test_issue_285(self):
+    def test_issue_285_2(self):
         """Test designed to check if the tool is robust against repositories with empty license """
+        github_data = Result()
         readme_text, github_data = process_files.process_repository_files(test_data_repositories + "RDFChess",
-                                                                                  {},
+                                                                                  github_data,
                                                                                   constants.RepositoryType.LOCAL)
-        assert 'license' not in github_data
+        assert 'license' not in github_data.results
 
     def test_no_repository_metadata(self):
         """Test designed to assess repositories with no releases"""
-        github_data = process_repository.load_online_repository_metadata(
-            "https://github.com/RDFLib/rdflib/tree/6.0.2")
-        assert 'releases' not in github_data
+        github_data, owner, repo_name, default_branch = process_repository.load_online_repository_metadata(Result(),
+            "https://github.com/oeg-upm/delta-ontology")
+        assert constants.CAT_RELEASES not in github_data.results.keys()
 
     def test_issue_284_issue_272(self):
         """Test designed to check if there are errors detecting title or stargazers"""
         github_data, owner, repo_name, default_br = process_repository.\
-            load_online_repository_metadata("https://github.com/3b1b/manim")
-        assert (('stargazersCount' in github_data) and ('longTitle' not in github_data))
+            load_online_repository_metadata(Result(), "https://github.com/3b1b/manim")
+        result_keys = github_data.results.keys()
+        assert ((constants.CAT_STARS in result_keys) and (constants.CAT_FULL_TITLE not in result_keys))
 
     def test_feature_462(self):
         """Test designed to process a repo and keep the results in disk"""
@@ -127,7 +152,8 @@ def test_feature_477(self):
         """
         Test designed to assess if readme.rst files are processed properly
         """
-        text, github_data = process_files.process_repository_files(test_data_repositories + "mir_eval", {},
+        github_data = Result()
+        text, github_data = process_files.process_repository_files(test_data_repositories + "mir_eval", github_data,
                                                                    constants.RepositoryType.LOCAL)
         assert len(text) > 0
 
@@ -135,7 +161,8 @@ def test_feature_477_2(self):
         """
         Test designed to assess if readme.txt files are processed properly
         """
-        text, github_data = process_files.process_repository_files(test_data_repositories + "resolver_deco", {},
+        github_data = Result()
+        text, github_data = process_files.process_repository_files(test_data_repositories + "resolver_deco", github_data,
                                                                    constants.RepositoryType.LOCAL)
         assert len(text) > 0
 
@@ -143,6 +170,7 @@ def test_feature_466(self):
         """
         Test designed to assess if a readme with non utf-8 encoding can be processed successfully
         """
-        text, github_data = process_files.process_repository_files(test_data_repositories + "corpuser", {},
+        github_data = Result()
+        text, github_data = process_files.process_repository_files(test_data_repositories + "corpuser", github_data,
                                                                    constants.RepositoryType.LOCAL)
         assert len(text) > 0
\ No newline at end of file
diff --git a/src/somef/test/test_process_results.py b/src/somef/test/test_process_results.py
new file mode 100644
index 00000000..05a2d9aa
--- /dev/null
+++ b/src/somef/test/test_process_results.py
@@ -0,0 +1,23 @@
+import unittest
+from ..utils import constants
+from ..process_results import Result
+
+
+class TestCli(unittest.TestCase):
+
+    def test_add_result(self):
+        # This test checks if results are correctly aggregated in their corresponding category
+        a = Result()
+        a.add_result("description", {"value":"This is a short desc", "type":"string"}, 0.9, constants.TECHNIQUE_SUPERVISED_CLASSIFICATION, "README.md")
+        a.add_result("description", {"value":"This is a long description", "type":"string"}, 0.8, constants.TECHNIQUE_REGULAR_EXPRESSION)
+        assert len(a.results["description"]) == 2
+
+    def test_add_result_2(self):
+        # This test checks if results that are incorrect (i.e., don't have type and value) get added
+        a = Result()
+        a.add_result("description", {"value":"This is a short desc", "type":"string"}, 0.9, constants.TECHNIQUE_SUPERVISED_CLASSIFICATION, "README.md")
+        # incorrect result (no type) below
+        a.add_result("description", {"value":"This is a short desc"}, 0.8, constants.TECHNIQUE_REGULAR_EXPRESSION)
+        assert len(a.results["description"]) == 1
+
+    # Test 3: category should be among the accepted categories. TO DO if time
\ No newline at end of file
diff --git a/src/somef/test/test_regular_expressions.py b/src/somef/test/test_regular_expressions.py
index cf0e695c..a7108f26 100644
--- a/src/somef/test/test_regular_expressions.py
+++ b/src/somef/test/test_regular_expressions.py
@@ -3,6 +3,8 @@
 from pathlib import Path
 
 from .. import regular_expressions
+from ..process_results import Result
+from ..utils import constants
 
 test_data_path = str(Path(__file__).parent / "test_data") + os.path.sep
 test_data_repositories = str(Path(__file__).parent / "test_data" / "repositories") + os.path.sep
@@ -14,165 +16,218 @@ def test_extract_bibtex(self):
         """Test designed to check if bibtext citations are detected"""
         with open(test_data_path + "test_extract_bibtex.txt", "r") as data_file:
             test_text = data_file.read()
-            c = regular_expressions.extract_bibtex(test_text)
-            assert "@inproceedings" in c[0]
+            c = regular_expressions.extract_bibtex(test_text, Result(), test_data_path + "test_extract_bibtex.txt")
+            result = c.results[constants.CAT_CITATION][0]
+            assert "@inproceedings" in result["result"]["value"]
 
     def test_extract_dois(self):
         """Test designed to check if doi links are detected"""
         with open(test_data_path + "test_extract_dois.txt", "r") as data_file:
             test_text = data_file.read()
-            c = regular_expressions.extract_dois(test_text)
-            assert len(c) == 2
+            c = regular_expressions.extract_doi_badges(test_text, Result(), test_data_path + "test_extract_dois.txt")
+            assert len(c.results[constants.CAT_IDENTIFIER]) == 2
 
     def test_extract_binder_links(self):
         """Test designed to check if binder links are detected"""
         with open(test_data_path + "test_extract_binder_links.txt", "r") as data_file:
             test_text = data_file.read()
-            c = regular_expressions.extract_binder_links(test_text)
-            assert len(c) == 2
+            c = regular_expressions.extract_binder_links(test_text, Result(),
+                                                         test_data_path + "test_extract_binder_links.txt")
+            assert len(c.results[constants.CAT_EXECUTABLE_EXAMPLE]) == 2
 
     def test_extract_title_underline(self):
         """Test designed to check if titles with underline markdown notation are detected"""
         with open(test_data_path + "test_extract_title_underline.txt", "r") as data_file:
             test_text = data_file.read()
-            c = regular_expressions.extract_title(test_text)
-            assert "Taguette" == c
+            c = regular_expressions.extract_title(test_text, Result(),
+                                                  test_data_path + "test_extract_title_underline.txt")
+            res = c.results[constants.CAT_FULL_TITLE][0]
+            assert "Taguette" == res[constants.PROP_RESULT][constants.PROP_VALUE]
 
     def test_extract_title_hash(self):
         """Test designed to check if titles with hash notation are detected"""
         with open(test_data_path + "test_extract_title_hash.txt", "r") as data_file:
             test_text = data_file.read()
-            c = regular_expressions.extract_title(test_text)
-            assert "T2WML: A Cell-Based Language To Map Tables Into Wikidata Records" == c
+            c = regular_expressions.extract_title(test_text, Result(), test_data_path + "test_extract_title_hash.txt")
+            res = c.results[constants.CAT_FULL_TITLE][0]
+            assert "T2WML: A Cell-Based Language To Map Tables Into Wikidata Records" == res[constants.PROP_RESULT][
+                constants.PROP_VALUE]
 
     def test_extract_title_with_md(self):
         """Test designed to check if titles are detected"""
         with open(test_data_path + "test_extract_title_with_md.txt", "r") as data_file:
             test_text = data_file.read()
-            c = regular_expressions.extract_title(test_text)
-            # print(c)
-            assert "SOMEF" == c
+            c = regular_expressions.extract_title(test_text, Result(),
+                                                  test_data_path + "test_extract_title_with_md.txt")
+            res = c.results[constants.CAT_FULL_TITLE][0]
+            assert "SOMEF" == res[constants.PROP_RESULT][constants.PROP_VALUE]
 
     def test_extract_readthedocs_1(self):
         """Test designed to check if readthedocs links are detected"""
         with open(test_data_path + "test_extract_readthedocs_1.txt", "r") as data_file:
             test_text = data_file.read()
-            c = regular_expressions.extract_readthedocs(test_text)
-            assert ["https://oba.readthedocs.io/"] == c
+            c = regular_expressions.extract_readthedocs(test_text, Result(),
+                                                        test_data_path + "test_extract_readthedocs_1.txt")
+            result = c.results[constants.CAT_DOCUMENTATION][0]
+            assert "https://oba.readthedocs.io/" == result[constants.PROP_RESULT][constants.PROP_VALUE]
+
+    def test_extract_readthedocs_1_1(self):
+        """Test to check if a result is added with a different name, then it gets added as a related link"""
+        with open(test_data_path + "test_extract_readthedocs_1.txt", "r") as data_file:
+            test_text = data_file.read()
+            r = Result()
+            r.add_result(constants.CAT_NAME, {
+                constants.PROP_TYPE: constants.STRING,
+                constants.PROP_VALUE: "unrelated"
+            }, 1, test_data_path + "test_extract_readthedocs_1.txt")
+            c = regular_expressions.extract_readthedocs(test_text, r, test_data_path + "test_extract_readthedocs_1.txt")
+            result = c.results[constants.CAT_RELATED_DOCUMENTATION][0]
+            assert "https://oba.readthedocs.io/" == result[constants.PROP_RESULT][constants.PROP_VALUE]
 
     def test_extract_readthedocs_2(self):
         """Test designed to check if readthedocs links are detected"""
         with open(test_data_path + "test_extract_readthedocs_2.txt", "r") as data_file:
             test_text = data_file.read()
-            c = regular_expressions.extract_readthedocs(test_text)
-            # print(c)
-            assert ["https://kgtk.readthedocs.io/"] == c
+            c = regular_expressions.extract_readthedocs(test_text, Result(),
+                                                        test_data_path + "test_extract_readthedocs_2.txt")
+            result = c.results[constants.CAT_DOCUMENTATION][0]
+            assert "https://kgtk.readthedocs.io/" == result[constants.PROP_RESULT][constants.PROP_VALUE]
 
     def test_extract_readthedocs_3(self):
         """Test designed to check if readthedoc links are detected"""
         test_text = """
         See full documentation at [https://somef.readthedocs.io/en/latest/](https://somef.readthedocs.io/en/latest/)
         """
-        c = regular_expressions.extract_readthedocs(test_text)
-        print(c)
-        assert ["https://somef.readthedocs.io/"] == c
+        c = regular_expressions.extract_readthedocs(test_text, Result(), test_data_path + "invented_path.txt")
+        result = c.results[constants.CAT_DOCUMENTATION][0]
+        assert "https://somef.readthedocs.io/" == result[constants.PROP_RESULT][constants.PROP_VALUE]
 
     def test_extract_readthedocs_issue_407(self):
         """Test designed to check if readthedocs links are detected"""
         with open(test_data_path + "test_extract_readthedocs_3.txt", "r") as data_file:
             test_text = data_file.read()
-            c = regular_expressions.extract_readthedocs(test_text)
-            print(c)
-            assert ["https://owl-to-oas.readthedocs.io/"] == c
+            c = regular_expressions.extract_readthedocs(test_text, Result(),
+                                                        test_data_path + "test_extract_readthedocs_3.txt")
+            result = c.results[constants.CAT_DOCUMENTATION][0]
+            assert "https://owl-to-oas.readthedocs.io/" == result[constants.PROP_RESULT][constants.PROP_VALUE]
 
     def test_extract_gitter_chat(self):
         """Test designed to check if gitter chats are detected"""
         with open(test_data_path + "test_extract_gitter_chat.txt", "r") as data_file:
             test_text = data_file.read()
-            c = regular_expressions.extract_support_channels(test_text)
-            # print(c)
-            assert "https://gitter.im/OpenGeoscience/geonotebook" in c
+            channels = regular_expressions.extract_support_channels(test_text, Result(),
+                                                                    test_data_path + "test_extract_gitter_chat.txt")
+            result = channels.results[constants.CAT_SUPPORT_CHANNELS]
+            assert "https://gitter.im/OpenGeoscience/geonotebook" == result[0][constants.PROP_RESULT][
+                constants.PROP_VALUE]
 
     def test_repo_status(self):
         """Test designed to check if repostatus badges are detected"""
         with open(test_data_path + "test_repo_status.txt", "r") as data_file:
             test_text = data_file.read()
-            repo_status = regular_expressions.extract_repo_status(test_text)
-            assert len(repo_status) > 0
+            repo_status = regular_expressions.extract_repo_status(test_text, Result(),
+                                                                  test_data_path + "test_repo_status.txt")
+            result = repo_status.results[constants.CAT_STATUS]
+            assert len(result) > 0 and result[0][constants.PROP_RESULT][
+                constants.PROP_VALUE] == "https://www.repostatus.org/#active"
 
     def test_issue_291(self):
         """Test designed to check if logos are detected"""
         repo_url = "https://github.com/dgarijo/Widoco"
         with open(test_data_path + "README-widoco.md", "r") as data_file:
             test_text = data_file.read()
-            logo, images = regular_expressions.extract_images(test_text, repo_url, None)
-            assert (not logo == "")
+            results = regular_expressions.extract_images(test_text, repo_url, None, Result(),
+                                                         test_data_path + "README-widoco.md", "master")
+            logo = results.results[constants.CAT_LOGO]
+            assert (logo[0][constants.PROP_RESULT][
+                        constants.PROP_VALUE] == "https://raw.githubusercontent.com/dgarijo/Widoco/master/src/main/resources/logo/logo2.png")
 
     def test_issue_291_2(self):
         """Test designed to check if logos are detected"""
         repo_url = "https://github.com/usc-isi-i2/kgtk/"
         with open(test_data_path + "test_logo_uscisii2.txt", "r") as data_file:
             test_text = data_file.read()
-            logo, images = regular_expressions.extract_images(test_text, repo_url, None)
-            assert (not logo == "")
+            results = regular_expressions.extract_images(test_text, repo_url, None, Result(),
+                                                         test_data_path + "test_logo_uscisii2.txt", "master")
+            logo = results.results[constants.CAT_LOGO]
+            assert (logo[0][constants.PROP_RESULT][
+                        constants.PROP_VALUE] == "https://github.com/usc-isi-i2/kgtk/raw/master/docs/images/kgtk_logo_200x200.png")
 
     def test_issue_291_3(self):
         """Test designed to check if logos are detected"""
         repo_url = "https://github.com/tensorflow/tensorflow/"
         with open(test_data_path + "test_logo_tensorflow.txt", "r") as data_file:
             test_text = data_file.read()
-            logo, images = regular_expressions.extract_images(test_text, repo_url, None)
-            assert (not logo == "")
+            results = regular_expressions.extract_images(test_text, repo_url, None, Result(),
+                                                         test_data_path + "test_logo_tensorflow.txt", "main")
+            logo = results.results[constants.CAT_LOGO]
+            assert (logo[0][constants.PROP_RESULT][
+                        constants.PROP_VALUE] == "https://www.tensorflow.org/images/tf_logo_horizontal.png")
 
     def test_issue_images(self):
         """Test designed to check if images are detected"""
         repo_url = "https://github.com/usc-isi-i2/kgtk/"
         with open(test_data_path + "test_issue_images.txt", "r") as data_file:
             test_text = data_file.read()
-            logo, images = regular_expressions.extract_images(test_text, repo_url, None)
-            assert len(images) > 0
-
-    def test_issue_181(self):
-        """Test designed to check if arxiv papers are detected"""
-        with open(test_data_path + "test_issue_181.txt", "r") as data_file:
-            test_text = data_file.read()
-            arxiv_links = regular_expressions.extract_arxiv_links(test_text)
-            assert len(arxiv_links) > 0
+            results = regular_expressions.extract_images(test_text, repo_url, None, Result(),
+                                                         test_data_path + "test_issue_images.txt", "main")
+            img = results.results[constants.CAT_IMAGE]
+            print(img)
+            assert len(img) == 2
+
+    # Test commented out because arxiv links with no context has demonstrated not to be useful
+    # def test_issue_181(self):
+    #     """Test designed to check if arxiv papers are detected"""
+    #     with open(test_data_path + "test_issue_181.txt", "r") as data_file:
+    #         test_text = data_file.read()
+    #         arxiv_links = regular_expressions.extract_arxiv_links(test_text)
+    #         assert len(arxiv_links) > 0
 
     def test_issue_270(self):
         """Test designed to check if support channels are detected"""
         with open(test_data_path + "test_issue_270.txt", "r") as data_file:
             test_text = data_file.read()
-            support_channels = regular_expressions.extract_support_channels(test_text)
-            assert len(support_channels) == 2
+            support_channels = regular_expressions.extract_support_channels(test_text, Result(),
+                                                                            test_data_path + "test_issue_270.txt")
+            assert len(support_channels.results[constants.CAT_SUPPORT_CHANNELS]) == 2
 
     def test_logo(self):
         """Test designed to check if logos are detected"""
         with open(test_data_path + "test_logo.txt", "r") as data_file:
             test_text = data_file.read()
-            logo, images = regular_expressions.extract_images(test_text, "https://github.com/oeg-upm/Chowlk", None)
-            assert (not logo == "")
+            results = regular_expressions.extract_images(test_text, "https://github.com/oeg-upm/Chowlk", None, Result(),
+                                                         test_data_path + "test_logo.txt", "master")
+            logo = results.results[constants.CAT_LOGO]
+            assert (logo[0][constants.PROP_RESULT][
+                        constants.PROP_VALUE] == "https://raw.githubusercontent.com/oeg-upm/Chowlk/webservice/resources/logo.png")
 
     def test_logo2(self):
         """Test designed to check if logos are detected"""
         with open(test_data_path + "test_logo2.txt", "r") as data_file:
             test_text = data_file.read()
-            logo, images = regular_expressions.extract_images(test_text, "https://github.com/pytorch/pytorch", None)
-            assert (not logo == "")
+            results = regular_expressions.extract_images(test_text, "https://github.com/pytorch/pytorch", None, Result(),
+                                                         test_data_path + "test_logo2.txt", "master")
+            logo = results.results[constants.CAT_LOGO]
+            assert (logo[0][constants.PROP_RESULT][
+                        constants.PROP_VALUE] == "https://raw.githubusercontent.com/pytorch/pytorch/master/docs/source/_static/img/pytorch-logo-dark.png")
 
     def test_images(self):
         """Test designed to check if images are detected"""
         with open(test_data_path + "test_images.txt", "r") as data_file:
             test_text = data_file.read()
-            logo, images = regular_expressions.extract_images(test_text, "https://github.com/pytorch/pytorch", None)
-            assert (len(images) > 0 and not logo == "")
+            results = regular_expressions.extract_images(test_text, "https://github.com/pytorch/pytorch", None, Result(),
+                                                         test_data_path + "test_images.txt", "master")
+            images = results.results[constants.CAT_IMAGE]
+            assert len(images) > 0
 
     def test_issue_320(self):
         """Test designed to check if logos are detected"""
         with open(test_data_path + "README-urllib3.md", "r") as data_file:
             test_text = data_file.read()
-            logo, images = regular_expressions.extract_images(test_text, "https://github.com/urllib3/urllib3", None)
-            assert (not logo == "")
+            results = regular_expressions.extract_images(test_text, "https://github.com/urllib3/urllib3", None, Result(),
+                                                         test_data_path + "README-urllib3.md", "master")
+            logo = results.results[constants.CAT_LOGO]
+            assert len(logo) > 0
 
     def test_issue_337(self):
         """Test designed to check if links are removed from the text sent to classifiers"""
@@ -191,47 +246,95 @@ def test_issue_337(self):
     def test_issue_427(self):
         with open(test_data_repositories + "Widoco" + os.path.sep + "README.md", "r") as data_file:
             test_text = data_file.read()
-            logo, images = regular_expressions.extract_images(test_text, None, test_data_repositories + "Widoco")
-            assert (logo.find('test_data') > 0)
+            results = regular_expressions.extract_images(test_text, None, test_data_repositories + "Widoco", Result(),
+                                                         test_data_repositories + "Widoco" + os.path.sep + "README.md",
+                                                         "master")
+            logo = results.results[constants.CAT_LOGO]
+            assert (logo[0][constants.PROP_RESULT][constants.PROP_VALUE].find('test_data') > 0)
 
     def test_issue_446(self):
         """Test that jitpack URLs are not returned"""
         with open(test_data_path + "README-licensius.md", "r") as data_file:
             test_text = data_file.read()
-            logo, images = regular_expressions.extract_images(test_text, "https://github.com/oeg-upm/licensius/", None)
-            assert (logo == "")
+            results = regular_expressions.extract_images(test_text, "https://github.com/oeg-upm/licensius/", None, Result(),
+                                                         test_data_repositories + "Widoco" + os.path.sep + "README.md",
+                                                         "master")
+            logo_results = results.results
+            assert (constants.CAT_LOGO not in logo_results.keys())
 
     def test_issue_446_1(self):
         """Test that jitpack URLs are not returned"""
         with open(test_data_path + "README-PPool.md", "r") as data_file:
             test_text = data_file.read()
-            logo, images = regular_expressions.extract_images(test_text, "https://github.com/oeg-upm/PPool/", None)
-            assert (logo == "" and len(images) == 0)
+            results = regular_expressions.extract_images(test_text, "https://github.com/oeg-upm/PPool/", None, Result(),
+                                                         test_data_path + "README-PPool.md", "master")
+            all_results = results.results
+            assert (constants.CAT_LOGO not in all_results.keys() and constants.CAT_IMAGE not in all_results.keys())
 
     def test_issue_446_2(self):
         """Test that jitpack URLs are not returned"""
         with open(test_data_path + "README-loupe-api.md", "r") as data_file:
             test_text = data_file.read()
-            logo, images = regular_expressions.extract_images(test_text, "https://github.com/oeg-upm/PPool/", None)
-            assert (logo == "" and len(images) == 1)
+            results = regular_expressions.extract_images(test_text, "https://github.com/oeg-upm/PPool/", None, Result(),
+                                                         test_data_path + "README-loupe-api.md", "master")
+            all_results = results.results
+            assert (constants.CAT_LOGO not in all_results.keys() and len(all_results[constants.CAT_IMAGE]) == 1)
 
     def test_issue_467(self):
         """Test designed to check if logo is correctly obtained"""
         with open(test_data_path + "README-wothive.md", "r") as data_file:
             test_text = data_file.read()
-            logo, images = regular_expressions.extract_images(test_text,"https://github.com/oeg-upm/wot-hive/", None)
-            assert logo == "https://raw.githubusercontent.com/oeg-upm/wot-hive/AndreaCimminoArriaga-wothive-logo/logo.png"
+            results = regular_expressions.extract_images(test_text, "https://github.com/oeg-upm/wot-hive/", None, Result(),
+                                                         test_data_path + "README-wothive.md", "master")
+            logo_results = results.results[constants.CAT_LOGO]
+            assert logo_results[0][constants.PROP_RESULT][constants.PROP_VALUE] == \
+                   "https://raw.githubusercontent.com/oeg-upm/wot-hive/AndreaCimminoArriaga-wothive-logo/logo.png"
 
     def test_issue_463(self):
         """Test designed to check if a repo with a badge but not logo is correctly ignored"""
         with open(test_data_path + "README-OBA-SPARQL.md", "r") as data_file:
             test_text = data_file.read()
-            logo, images = regular_expressions.extract_images(test_text,"https://github.com/KnowledgeCaptureAndDiscovery/OBA_sparql/", None)
-            assert logo == ""
+            results = regular_expressions.extract_images(test_text,
+                                                              "https://github.com/KnowledgeCaptureAndDiscovery/OBA_sparql/",
+                                                              None, Result(), test_data_path + "README-OBA-SPARQL.md", "master")
+            all_results = results.results
+            assert constants.CAT_LOGO not in all_results.keys()
 
     def test_issue_463_2(self):
         """Test designed to check if a repo with a badge but not logo is correctly ignored"""
         with open(test_data_path + "README-base-images.md", "r") as data_file:
             test_text = data_file.read()
-            logo, images = regular_expressions.extract_images(test_text,"https://github.com/mintproject/base_images", None)
-            assert logo == ""
+            results = regular_expressions.extract_images(test_text, "https://github.com/mintproject/base_images",
+                                                              None, Result(), test_data_path + "README-base-images.md", "master")
+            all_results = results.results
+            assert constants.CAT_LOGO not in all_results.keys()
+
+    def test_wiki(self):
+        """Test designed to check if a repository with wiki links retrieves the wiki links as documentation"""
+        with open(test_data_path + "test_wiki.md", "r") as data_file:
+            test_text = data_file.read()
+            repo_status = regular_expressions.extract_wiki_links(test_text, "https://github.com/oeg-upm/mapeathor",
+                                                                 Result(), test_data_path + "test_wiki.md")
+            result = repo_status.results[constants.CAT_DOCUMENTATION]
+            assert len(result) > 0 and result[0][constants.PROP_RESULT][
+                constants.PROP_VALUE] == "https://github.com/oeg-upm/Mapeathor/wiki"
+
+    def test_package_distribution(self):
+        """Test designed to check if a repository with pypi links is properly recovered"""
+        with open(test_data_path + "README-mapeathor.md", "r") as data_file:
+            test_text = data_file.read()
+            repo_status = regular_expressions.extract_package_distributions(test_text, Result(),
+                                                                            test_data_path + "README-mapeathor.md")
+            result = repo_status.results[constants.CAT_PACKAGE_DISTRIBUTION]
+            assert len(result) > 0 and result[0][constants.PROP_RESULT][
+                constants.PROP_VALUE] == "https://pypi.org/project/mapeathor/"
+
+    def test_package_distribution_2(self):
+        """Test designed to check if a repository with pypi links is properly recovered"""
+        with open(test_data_path + "test_pypi.md", "r") as data_file:
+            test_text = data_file.read()
+            repo_status = regular_expressions.extract_package_distributions(test_text, Result(),
+                                                                            test_data_path + "test_pypi.md")
+            result = repo_status.results[constants.CAT_PACKAGE_DISTRIBUTION]
+            assert len(result) > 0 and "https://pypi.org/project/inspect4py" in result[0][constants.PROP_RESULT][
+                constants.PROP_VALUE]
diff --git a/src/somef/test/test_supervised_classification.py b/src/somef/test/test_supervised_classification.py
index eea36e2e..8c4367f9 100644
--- a/src/somef/test/test_supervised_classification.py
+++ b/src/somef/test/test_supervised_classification.py
@@ -2,17 +2,20 @@
 import os
 from pathlib import Path
 from .. import supervised_classification
+from ..process_results import Result
+from ..utils import constants
 
 # Test data for tests
 test_data_path = str(Path(__file__).parent / "test_data") + os.path.sep
 
 
-class TestRegExp(unittest.TestCase):
+class TestSupervisedClassification(unittest.TestCase):
 
     def test_run_category_classification(self):
         """Test to check if category classification runs correctly."""
         with open(test_data_path + "README-widoco.md", "r") as data_file:
             text = data_file.read()
-            categories = supervised_classification.run_category_classification(text, 0.8)
-            self.assertEqual(len(categories), 1)
-            self.assertEqual(categories[0]['value'], ["Semantic web"])
+            result = supervised_classification.run_category_classification(text, 0.8, Result())
+            self.assertEqual(len(result.results[constants.CAT_APPLICATION_DOMAIN]), 1)
+            cat_result = result.results[constants.CAT_APPLICATION_DOMAIN][0]
+            self.assertEqual(cat_result[constants.PROP_RESULT]['value'], "Semantic web")