Repository: KnowledgeCaptureAndDiscovery/somef (language: python)

Issue:
If super header is installation instructions, current header may also be
See https://github.com/oeg-upm/easytv-annotator
Installation instructions is incorrect because "The hard way" and "The easy way" are not tagged as installation instructions.

I think that if a parent header is "Installation", then this header should be installation too.
Error when extracting header
Using somef with https://github.com/oeg-upm/fair_ontologies

There is a problem extracting the header because it extracts up to the previous version of "Demo" in the readme. Bug with the parser.

Expected behavior: 
Extract `A public demo of FOOPS! is available here: https://w3id.org/foops/`

Current behavior: 
```
 publisher    = {CEUR-WS.org},\n    series       = {{CEUR} Workshop Proceedings},\n    volume       = 2980,\n    url          = {http://ceur-ws.org/Vol-2980/paper321.pdf}\n}\n```\nIf you are interested in more information, check [our slides](https://www.slideshare.net/dgarijo/foops-an-ontology-pitfall-scanner-for-the-fair-principles) with the rationale of FOOPS! and the [teaser video](https://www.youtube.com/watch?v=s8FaFl8i6yQ&ab_channel=OEG-UPM) we made for ISWC 2021.\n\nThe client application has been integrated from the work in https://github.com/jacobomata/FairValidator. The client would not have been possible without the work from @jacobomata\n\n## Demo\nA public demo of FOOPS! is available here: [https://w3id.org/foops/](https://w3id.org/foops/)\n
```


Discussion / hints:

Fix patch:
diff --git a/src/somef/cli.py b/src/somef/cli.py
index e95f608a..9a75c402 100644
--- a/src/somef/cli.py
+++ b/src/somef/cli.py
@@ -980,11 +980,12 @@ def create_excerpts(string_list):
     divisions = parser_somef.extract_blocks_excerpts(string_list)
     print("Text Successfully split. \n")
     # issue 337
-    output = []
+    output = {}
     for division in divisions:
+        original = division
         division = remove_links_images(division)
         if len(division) > 0:
-            output.append(division)
+            output[division] = original
     return output
 
 
@@ -1003,6 +1004,11 @@ def run_classifiers(excerpts, file_paths):
     """
     score_dict = {}
     if len(excerpts) > 0:
+        text_to_classifier = []
+        text_to_results = []
+        for key in excerpts.keys():
+            text_to_classifier.append(key)
+            text_to_results.append(excerpts[key])
         for category in categories:
             if category not in file_paths.keys():
                 sys.exit("Error: Category " + category + " file path not present in config.json")
@@ -1011,8 +1017,8 @@ def run_classifiers(excerpts, file_paths):
                 sys.exit(f"Error: File/Directory {file_name} does not exist")
             print("Classifying excerpts for the category", category)
             classifier = pickle.load(open(file_name, 'rb'))
-            scores = classifier.predict_proba(excerpts)
-            score_dict[category] = {'excerpt': excerpts, 'confidence': scores[:, 1]}
+            scores = classifier.predict_proba(text_to_classifier)
+            score_dict[category] = {'excerpt': text_to_results, 'confidence': scores[:, 1]}
             print("Excerpt Classification Successful for the Category", category)
         print("\n")
 
@@ -1046,6 +1052,15 @@ def remove_unimportant_excerpts(excerpt_element):
     return final_excerpt
 
 
+def is_in_excerpts_headers(text, set_excerpts):
+    set_text = set(text.split())
+    for excerpt in set_excerpts:
+        set_excerpt = set(excerpt.split())
+        if set_text.issubset(set_excerpt):
+            return True, excerpt
+
+    return False, None
+
 ## Function takes scores dictionary and a threshold as input
 ## Returns predictions containing excerpts with a confidence above the given threshold.
 def classify(scores, threshold, excerpts_headers, header_parents):
@@ -1061,31 +1076,48 @@ def classify(scores, threshold, excerpts_headers, header_parents):
         for i in range(len(scores[ele]['confidence'])):
             if scores[ele]['confidence'][i] >= threshold:
                 element = scores[ele]['excerpt'][i]
-                if element in set(excerpts_headers['text']):
-                    elem = excerpts_headers.loc[excerpts_headers['text'] == element]
-                    ind = elem.index.values[0]
-                    header = elem.at[ind, 'header']
-                if flag == False:
+                # if excerpt is empty, it means it's the first iteration of the loop
+                if excerpt == "":
+                    if element in set(excerpts_headers['text']):
+                        elem = excerpts_headers.loc[excerpts_headers['text'] == element]
+                        ind = elem.index.values[0]
+                        header = elem.at[ind, 'header']
                     excerpt = excerpt + scores[ele]['excerpt'][i] + ' \n'
                     confid.append(scores[ele]['confidence'][i])
-                    flag = True
                 else:
-                    excerpt = excerpt + scores[ele]['excerpt'][i] + ' \n'
-                    confid.append(scores[ele]['confidence'][i])
-            else:
-                if flag == True:
-                    if not header == "":
-                        element = remove_unimportant_excerpts(
-                            {'excerpt': excerpt, 'confidence': confid, 'originalHeader': header,
-                             'parentHeader': header_parents[header]})
-                        header == ""
+                    current_header = ""
+                    if element in set(excerpts_headers['text']):
+                        elem = excerpts_headers.loc[excerpts_headers['text'] == element]
+                        ind = elem.index.values[0]
+                        current_header = elem.at[ind, 'header']
+                    # if both headers are the same, the new data is added
+                    if header == current_header:
+                        excerpt = excerpt + scores[ele]['excerpt'][i] + ' \n'
+                        confid.append(scores[ele]['confidence'][i])
+                    # if they are not the same, a new excerpt is created with the previous data
+                    # and store the new data as part of a new excerpt
                     else:
-                        element = remove_unimportant_excerpts({'excerpt': excerpt, 'confidence': confid})
-                    if len(element['confidence']) != 0:
-                        predictions[ele].append(element)
-                    excerpt = ""
-                    confid = []
-                    flag = False
+                        if not header == "":
+                            element = remove_unimportant_excerpts(
+                                {'excerpt': excerpt, 'confidence': confid, 'originalHeader': header,
+                                 'parentHeader': header_parents[header]})
+                        else:
+                            element = remove_unimportant_excerpts({'excerpt': excerpt, 'confidence': confid})
+                        if len(element['confidence']) != 0:
+                            predictions[ele].append(element)
+                        header = current_header
+                        excerpt = scores[ele]['excerpt'][i] + ' \n'
+                        confid = [scores[ele]['confidence'][i]]
+        # if an element hasn't been added, it's added at this point
+        if excerpt != "":
+            if not header == "":
+                element = remove_unimportant_excerpts(
+                    {'excerpt': excerpt, 'confidence': confid, 'originalHeader': header,
+                     'parentHeader': header_parents[header]})
+            else:
+                element = remove_unimportant_excerpts({'excerpt': excerpt, 'confidence': confid})
+            if len(element['confidence']) != 0:
+                predictions[ele].append(element)
         print("Run completed.")
     print("All Excerpts below the given Threshold Removed. \n")
     return predictions
diff --git a/src/somef/parser_somef.py b/src/somef/parser_somef.py
index 66b7170e..cf98e009 100644
--- a/src/somef/parser_somef.py
+++ b/src/somef/parser_somef.py
@@ -15,7 +15,8 @@ def extract_headers(original_text):
         line = splitted[index]
         if line.startswith("<h"):
             if is_header(line):
-                text = re.sub(regex, '', line)
+                #text = re.sub(regex, '', line)
+                text = get_tag_content(line)
                 if index + 1 >= limit:
                     output[text] = True
                 elif not splitted[index + 1].startswith("<h"):
@@ -37,7 +38,7 @@ def extract_headers_with_tags(original_text):
         line = splitted[index]
         if line.startswith("<h"):
             if is_header(line):
-                output.append(line)
+                output.append(replace_html_tags(line))
         index += 1
     return output
 
@@ -85,12 +86,17 @@ def extract_content_per_header(original_text, headers):
 
 def get_position(init_index, text_tokenized, text):
     while init_index < len(text_tokenized):
-        val = text_tokenized[init_index].find(text)
-        if text_tokenized[init_index].find(text) != -1:
+        val = text_tokenized[init_index]
+        val = remove_hash(val).strip()
+        if val.startswith(text):
             return init_index
         init_index += 1
     return -1
 
+def remove_hash(text):
+    while text.startswith("#"):
+        text = text[1:]
+    return text
 
 def get_text(init_index, end_index, text_tokenized):
     if end_index == -1:
@@ -110,6 +116,8 @@ def extract_bash(text):
     while text.find('```')!= -1:
         init = text.find('```')
         end = text.find('```',init+3)
+        if end == -1:
+            break
         bash_text = text[init:end+3]
         text = text.replace(bash_text, "BASH"+str(index)+"*")
         output["BASH"+str(index)+"*"] = bash_text
@@ -416,4 +424,18 @@ def is_header(header):
             or header.startswith('<h4') or header.startswith('<h5') or header.startswith('<h6'):
         return True
     else:
-        return False
\ No newline at end of file
+        return False
+
+
+def get_tag_content(header):
+    init = header.index(">")
+    end = header.index("</h")
+    return replace_html_tags(header[init+1:end])
+
+
+def replace_html_tags(text):
+    text = text.replace("<i>","*").replace("</i>","*")
+    text = text.replace("<em>","*").replace("</em>","*")
+    text = text.replace("<b>", "**").replace("</b>", "**")
+    text = text.replace("<strong>", "**").replace("</strong>", "**")
+    return text
\ No newline at end of file


Test patch:
diff --git a/src/somef/test/test_cli.py b/src/somef/test/test_cli.py
index e0772652..dc6473e0 100644
--- a/src/somef/test/test_cli.py
+++ b/src/somef/test/test_cli.py
@@ -460,5 +460,66 @@ def test_issue_379(self):
         text_file.close()
         json_content = json.loads(data)
         description = json_content['description']
-        assert len(description) == 1
-        os.remove(test_data_path + "test-379.json")
\ No newline at end of file
+        assert len(description) > 0
+        os.remove(test_data_path + "test-379.json")
+
+    def test_issue_383(self):
+        run_cli(threshold=0.8,
+                ignore_classifiers=False,
+                repo_url=None,
+                doc_src=test_data_path + "README-mapeathor.md",
+                in_file=None,
+                output=test_data_path + "test-383.json",
+                graph_out=None,
+                graph_format="turtle",
+                codemeta_out=None,
+                pretty=True,
+                missing=True)
+        text_file = open(test_data_path + "test-383.json", "r")
+        data = text_file.read()
+        text_file.close()
+        assert data.find("- [Ana Iglesias-Molina](https://github.com/anaigmo) (ana.iglesiasm@upm.es)") > 0
+        os.remove(test_data_path + "test-383.json")
+
+    def test_issue_384(self):
+        run_cli(threshold=0.8,
+                ignore_classifiers=False,
+                repo_url=None,
+                doc_src=test_data_path + "README-fair-ontologies.md",
+                in_file=None,
+                output=test_data_path + "test-384.json",
+                graph_out=None,
+                graph_format="turtle",
+                codemeta_out=None,
+                pretty=True,
+                missing=True)
+        text_file = open(test_data_path + "test-384.json", "r")
+        data = text_file.read()
+        text_file.close()
+        assert data.find("A public demo of FOOPS! is available here:") > 0
+        os.remove(test_data_path + "test-384.json")
+
+    def test_issue_378(self):
+        run_cli(threshold=0.8,
+                ignore_classifiers=False,
+                repo_url=None,
+                doc_src=test_data_path + "README-easytv-annotator.md",
+                in_file=None,
+                output=test_data_path + "test-378.json",
+                graph_out=None,
+                graph_format="turtle",
+                codemeta_out=None,
+                pretty=True,
+                missing=True)
+        text_file = open(test_data_path + "test-378.json", "r")
+        data = text_file.read()
+        text_file.close()
+        json_content = json.loads(data)
+        installation = json_content['installation']
+        assert_true = False
+        for install in installation:
+            if 'parentHeader' in install.keys():
+                if install['originalHeader'] == "Hard way" and install['parentHeader'] == "Installation":
+                    assert_true = True
+        assert assert_true
+        os.remove(test_data_path + "test-378.json")
\ No newline at end of file
diff --git a/src/somef/test/test_data/README-easytv-annotator.md b/src/somef/test/test_data/README-easytv-annotator.md
new file mode 100644
index 00000000..76e075ac
--- /dev/null
+++ b/src/somef/test/test_data/README-easytv-annotator.md
@@ -0,0 +1,74 @@
+ # easytv-annotator
+Sign language annotator library for the EASYTV european project
+
+
+TRANSLATOR PROJECT
+
+## Prerequisites
+Project is linked to the easytv project resources: https://github.com/oeg-upm/easytv-resources
+
+It uses Perl for TreeTagger. Install utf8
+> perl -MCPAN -e shell
+> install utf8::all
+
+
+## Installation
+
+This project is compiled with maven and there are four libraries that are not available at maven central repo.
+These jar are in the /lib folder and must be added to your local maven repo. 
+
+### Easy way
+
+Using maven through the command install. The pom file contains all the instructions to install the dependencies
+
+```sh
+$ mvn install
+```
+
+### Hard way
+Through commands: The basic comand is 
+```sh
+$ mvn install:install-file -Dfile=${jarfile} -DgroupId=${group.id} -DartifactId=${lib.id} -Dversion=${version} -Dpackaging=jar
+```
+
+Example: 
+```sh
+$ cd lib
+$ mvn install:install-file -Dfile=lib/babelnet-api-3.7.1.jar -DgroupId=it.uniroma1.lcl.babelnet -DartifactId=babelnet-api -Dversion=3.7.1 -Dpackaging=jar
+```
+
+There is a .bat file and a .sh file for install all the dependencies: 
+- InstallDependencies.sh
+- InstallDependencies.bat
+
+
+
+
+## Build
+
+First build the project with maven.
+The compiled jar of the project is copied to the /dist folder, with all the necessary libraries.
+
+## Execution
+To execute the project you can work on your own IDE and run the class 'Execution' or execute the compiled jar as
+
+```sh
+$ cd dist
+$ java -jar EasyTranslator-1.0.jar
+```
+
+The execution provides a console in which you can write sentences specifiying their language. The program parses the sentence with CoreNLP and asks BabelNet synsets for Verbs and Nouns. Then, the program sends the sentence to BabelFly
+
+
+## Properties
+
+BabelNet and BabelFly need two specific folders in the root folder: config and resources.
+These two are repeated in the project root folder and in the dist folder. 
+
+
+Personal keys for accessing BabelNet and BabelFly are introduced in the files:
+- config/babelfy.var.properties
+- config/babelnet.var.properties
+
+
+
diff --git a/src/somef/test/test_data/README-fair-ontologies.md b/src/somef/test/test_data/README-fair-ontologies.md
new file mode 100644
index 00000000..2b22f5c7
--- /dev/null
+++ b/src/somef/test/test_data/README-fair-ontologies.md
@@ -0,0 +1,53 @@
+# OEG FAIR Ontologies Validator
+
+Authors: Daniel Garijo and María Poveda, with contributions from Jacobo Mata.
+
+FOOPS! is an application for validating whether a vocabulary (OWL or SKOS) conforms with the FAIR data principles.
+
+Our [ISWC 2021 demo paper](html_client/assets/iswc_2021_demo.pdf) (**best demo award**) provides an overview of the FOOPS! service. Please cite our work as follows:
+```
+@article{foops2021,
+    title        = {FOOPS!: An Ontology Pitfall Scanner for the FAIR Principles},
+    author       = {Garijo, Daniel and Corcho, Oscar and Poveda-Villal{\'o}n, Mar{\i}a},
+    year         = 2021,
+    booktitle    = {International Semantic Web Conference (ISWC) 2021: Posters, Demos, and Industry Tracks},
+    publisher    = {CEUR-WS.org},
+    series       = {{CEUR} Workshop Proceedings},
+    volume       = 2980,
+    url          = {http://ceur-ws.org/Vol-2980/paper321.pdf}
+}
+```
+If you are interested in more information, check [our slides](https://www.slideshare.net/dgarijo/foops-an-ontology-pitfall-scanner-for-the-fair-principles) with the rationale of FOOPS! and the [teaser video](https://www.youtube.com/watch?v=s8FaFl8i6yQ&ab_channel=OEG-UPM) we made for ISWC 2021.
+
+The client application has been integrated from the work in https://github.com/jacobomata/FairValidator. The client would not have been possible without the work from @jacobomata
+
+## Demo
+A public demo of FOOPS! is available here: [https://w3id.org/foops/](https://w3id.org/foops/)
+
+## Installation instructions
+The project was build and tested with JDK 11.0.11 in Ubuntu.
+
+To create the JAR, just run:
+
+```
+mvn install
+```
+on the server folder, or download the JAR from the releases page.
+
+## Running the server
+Run the following command
+
+```
+java -jar -Dserver.port=PORT fair_ontologies-0.0.1.jar
+```
+
+Where PORT is the port you want to run the server
+
+to test the installation, just do a curl command:
+
+```
+curl -X POST "http://localhost:8083/assessOntology" -H "accept: application/json;charset=UTF-8" -H "Content-Type: application/json;charset=UTF-8" -d "{ \"ontologyUri\": \"https://w3id.org/okn/o/sd\"}"
+```
+
+As a result, you should see a JSON in your console, such as the one in sample.json.
+
diff --git a/src/somef/test/test_data/README-mapeathor.md b/src/somef/test/test_data/README-mapeathor.md
new file mode 100644
index 00000000..1b8e81e6
--- /dev/null
+++ b/src/somef/test/test_data/README-mapeathor.md
@@ -0,0 +1,45 @@
+ [![Generic badge](https://img.shields.io/badge/Status-Developing-yellow)](https://shields.io/)
+ [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://github.com/oeg-upm/Mapeathor/blob/master/LICENSE)
+ [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.6025390.svg)](https://doi.org/10.5281/zenodo.6025390)
+ [![version](https://img.shields.io/badge/python-3.6+-blue.svg)](https://www.python.org/downloads/release/python-360/)
+ [![Latest PyPI version](https://img.shields.io/pypi/v/mapeathor?style=flat)](https://pypi.python.org/pypi/mapeathor)
+
+<p align="center">
+ <img src="https://raw.githubusercontent.com/oeg-upm/Mapeathor/master/imgs/Square_logo_mapeathor.png" alt="workflow" width="150"/>
+</p>
+
+
+# Mapeathor
+#### Mapeathor translates your mapping rules specified in spreadsheets to a mapping language.  
+
+[Mapeathor](https://morph.oeg.fi.upm.es/tool/mapeathor) is a simple spreadsheet parser able to generate mapping rules in three mapping languages: R2RML, RML (with extension to functions from FnO) and YARRRML. It takes the mapping rules expressed in a spreadsheet and transforms them into the desired language. The spreadsheet template is designed to facilitate the mapping rules' writting, with the aim of being language independent, and thus, lowering the barrier of generating mappings for non-expert users.
+
+<p align="center">
+ <img src="https://raw.githubusercontent.com/oeg-upm/Mapeathor/master/imgs/workflow.png" alt="workflow" width="600"/>
+</p>
+
+## Example
+A more detailed explanation is provided in the [wiki](https://github.com/oeg-upm/Mapeathor/wiki).
+
+### First Step: Fill the xlsx template with the transformation rules
+The template has five mandatory sheets, *Prefixes, Source, Subject PredicateObjectMap* and *Functions*. The last one can be left blank in case there are no functions. The spreadsheet can be in XLSX format or a Google Spreadsheet. Here is an example of the structure of the spreadsheet.
+
+<p align="center">
+ <img src="https://raw.githubusercontent.com/oeg-upm/Mapeathor/master/imgs/sheets.png" alt="sheets" width="500"/>
+</p>
+
+### Second Step: Choose the output language
+One of three options can be chosen: R2RML, RML or YARRRML.
+
+### Third Step: Run it!
+The easiest way of running Mapeathor is using the [web service](https://morph.oeg.fi.upm.es/demo/mapeathor) and the [Swagger](https://morph.oeg.fi.upm.es/tool/mapeathor/swagger/) instance. For CLI lovers, the service is available as a [PyPi package](https://pypi.org/project/mapeathor/) and Docker image. The instructions of the latest can be found in the [wiki](https://github.com/oeg-upm/Mapeathor/wiki).
+
+## Publications
+Iglesias-Molina, A., Pozo-Gilo, L., Dona, D., Ruckhaus, E., Chaves-Fraga, D., & Corcho, O. (2020, January). *Mapeathor: Simplifying the Specification of Declarative Rules for Knowledge Graph Construction. In ISWC (Demos/Industry).* [Online version](http://ceur-ws.org/Vol-2721/paper488.pdf)
+
+Iglesias-Molina, A., Chaves-Fraga, D., Priyatna, F., & Corcho, O. (2019). Towards the Definition of a Language-Independent Mapping Template for Knowledge Graph Creation. *In Proceedings of the Third International Workshop on Capturing Scientific Knowledge co-located with the 10th International Conference on Knowledge Capture (K-CAP 2019)* (pp. 33-36). [Online version](https://sciknow.github.io/sciknow2019/papers/SciKnow_2019_paper_4.pdf)
+
+## Authors and contact
+- [Ana Iglesias-Molina](https://github.com/anaigmo) (ana.iglesiasm@upm.es)
+- [Luis Pozo](https://github.com/w0xter) (luis.pozo@upm.es)
+- [Daniel Doña](https://github.com/daniel-dona) (daniel.dona@xnor.ga)
diff --git a/src/somef/test/test_data/README-raidologist.md b/src/somef/test/test_data/README-raidologist.md
index 67ce934d..a495f5cd 100644
--- a/src/somef/test/test_data/README-raidologist.md
+++ b/src/somef/test/test_data/README-raidologist.md
@@ -1,7 +1,5 @@
 # r.AID.ologist 🖊️
-
 ### 🔍**Description of the system**
-
 r.AID.ologist is a framework designed for radiologists to provide assistance on the generation of medical reports. 
 It is devised as a Case-Based Reasoning model, powered by several deep learning models which have been included to improve its performance, as well as providing additional features.  
 
@@ -10,24 +8,19 @@ While CBR is integrated as a continuous cycle, its subparts are designed to be f
 <img src="https://i.ibb.co/7Y4ZyhR/Global-Diagram.png" width=50% height=50%>
 
 ### 🧱**Module specification**
-
 The implementations of the different modules are contained in the *functions.py* file from the internal_functions folder. By default, the following models are employed for each of the denoted modules:
-
 #### Retrieve Modules
-
  1. ***Image feature extraction***: A combination of a keypoint-based image feature algorithm (KAZE) and a pretrained Convolutional Neural Network (ResNet18). 
  2. ***Document embedding***:  An English pretrained model from SciSpacy is used (en_core_sci_md)
  3. ***Named Entity Recognition***: An external resource, CliNER, is employed for this task. The employed pretrained model works for English textual data, and distinguishes between three types of named entities: problems, treatments and tests. These types **MUST** remain unchangeable in case of substitution. 
  4. ***Noise filtering***: SciSpacy is used to detect existing abbreviations. The ratio of identified abbreviations is obtained as *#_(detected_abbreviations ^ existing_tokens)/ #_detected abbreviations*
 
 #### Reuse Modules
-
  1. ***Sectioning Model***: A bidirectional LSTM, implemented via Pytorch, is used to section the input report. The employed sectioning model, alongside its functions, is implemented in the *section_model.py* file contained in the internal_functions folder. In this default model, four sections are considered: **Findings, Comparison, Indication and Impression**
  2. ***Scoring Model***: A combination of the aforementioned SciSpacy embedding model with a Random Forest Classifier is employed  to score each report as valid (1) or rejected (0).
  3. ***Abbreviation Disambiguation***: The aforementioned SciSpacy English pretrained model is used to extract existing abbreviations. SNOMED's query service is then used to obtain potential disambiguations for each of the detected abbreviations.
 
 ### 🖱️ **Installation**
-
 A ready-to-use version of the framework is available as a Docker image in **elviish/raidologist:latest**. Once downloaded, the image can be ran with the command:
 
     docker run --name raidologist -p *selected_port*:5000 elviish/raidologist:latest