Repository: asreview/asreview

Issue:
ASReview fails on RIS files with label tag
RIS files with numeric `LB` field fail on search and review. ASReview classification value is filled with those LB values. Because they are not `0` or `1`, items are handled incorrectly in the prior knowledge step (and elsewhere, but hard to get there).

Workaround: Remove `LB` fields from RIS file. 

```
TY  - JOUR
AB  - p53 functions to induce cellular senescence, which is incompatible with self-renewal of pluripotent stem cells such as induced pluripotent stem cells (iPSC) and embryonic stem cells (ESC). However, p53 also has essential roles in these cells through DNA damage repair for maintaining genomic integrity and high sensitivity to apoptosis for eliminating severely damaged cells. We hypothesized that "133p53, a physiological inhibitory p53 isoform, is involved in the balanced regulation of self-renewing capacity, DNA damage repair and apoptosis. We examined 12 lines of human iPSC and their original fibroblasts, as well as three ESC lines, for endogenous protein levels of "133p53 and full-length p53 (FL-p53), and mRNA levels of various p53 target genes. While FL-p53 levels in iPSC and ESC widely ranged from below to above those in the fibroblasts, all iPSC and ESC lines expressed elevated levels of "133p53. The p53-inducible genes that mediate cellular senescence (p21 WAF1, miR-34a, PAI-1 and IGFBP7), but not those for apoptosis (BAX and PUMA) and DNA damage repair (p53R2), were downregulated in iPSC and ESC. Consistent with these endogenous expression profiles, overexpression of "133p53 in human fibroblasts preferentially repressed the p53-inducible senescence mediators and significantly enhanced their reprogramming to iPSC. The iPSC lines derived from "133p53-overexpressing fibroblasts formed well-differentiated, benign teratomas in immunodeficient mice and had fewer numbers of somatic mutations than an iPSC derived from p53-knocked-down fibroblasts, suggesting that "133p53 overexpression is non-or less oncogenic and mutagenic than total inhibition of p53 activities. Overexpressed "133p53 prevented FL-p53 from binding to the regulatory regions of p21 WAF1 and miR-34a promoters, providing a mechanistic basis for its dominant-negative inhibition of a subset of p53 target genes. This study supports the hypothesis that upregulation of "133p53 is an endogenous mechanism that facilitates human somatic cells to become self-renewing pluripotent stem cells with maintained apoptotic and DNA repair activities.
AD  - C.C. Harris, Laboratory of Human Carcinogenesis, Center for Cancer Research, National Cancer Institute, National Institutes of Health, 37 Convent Drive, Bethesda, MD, United States
AN  - 28362428
AU  - Horikawa, I.
AU  - Park, K. Y.
AU  - Isogaya, K.
AU  - Hiyoshi, Y.
AU  - Li, H.
AU  - Anami, K.
AU  - Robles, A. I.
AU  - Mondal, A. M.
AU  - Fujita, K.
AU  - Serrano, M.
AU  - Harris, C. C.
DB  - Embase
Medline
DO  - 10.1038/cdd.2017.48
IS  - 6
KW  - DNA
isoprotein
messenger RNA
microRNA 34a
mitochondrial DNA
protein Bax
protein p53
PUMA protein
alternative RNA splicing
apoptosis
article
cell differentiation
chromatin immunoprecipitation
controlled study
DNA binding
DNA damage
DNA repair
DNA responsive element
embryo
fetus
fibroblast
gene frequency
gene overexpression
gene repression
gene targeting
genetic code
human
human cell
human embryonic stem cell
indel mutation
induced pluripotent stem cell
newborn
normal human
nuclear reprogramming
priority journal
promoter region
protein degradation
protein expression
selective autophagy
senescence
somatic mutation
stem cell self-renewal
transactivation
upregulation
LA  - English
LB  - 28362428
M3  - Article
N1  - Cell Death and Differentiation
L615137790
2017-04-07
2017-06-05
PY  - 2017
SN  - 1476-5403
1350-9047
SP  - 1017-1028
ST  - Δ133p53 represses p53-inducible senescence genes and enhances the generation of human induced pluripotent stem cells
T2  - Cell Death Differ
TI  - Δ133p53 represses p53-inducible senescence genes and enhances the generation of human induced pluripotent stem cells
UR  - https://www.embase.com/search/results?subaction=viewrecord&id=L615137790&from=export
VL  - 24
ID  - 2111
ER  - 

```

Hints from issue comments:
@J535D165: How to fix a project with incorrect labels:

1. make a backup of the project
2. open the project folder
3. open the RIS file and remove all lines with `LB -`
4. delete the `.pickle` file in the project's data folder
5.  open the SQL file in the reviews folder with a tool like https://sqlitebrowser.org/
6. remove all lines in the results table with label > 1. Save the changes
7. change the status in `project.json` to "review"@luukvanesveld: I've had the same issue and it was quickly fixed. Thank you!!


> How to fix a project with incorrect labels:
> 
> 1. make a backup of the project
> 2. open the project folder
> 3. open the RIS file and remove all lines with `LB -`
> 4. delete the `.pickle` file in the project's data folder
> 5. open the SQL file in the reviews folder with a tool like https://sqlitebrowser.org/
> 6. remove all lines in the results table with label > 1. Save the changes
> 7. change the status in `project.json` to "review"


@J535D165: @PeterLombaers, is this issue still present with the new data store?

Solution patch:
diff --git a/asreview/data/ris.py b/asreview/data/ris.py
index ebfa150d..dd22a8e1 100644
--- a/asreview/data/ris.py
+++ b/asreview/data/ris.py
@@ -210,6 +210,13 @@ class RISReader(BaseReader):
 
         return df
 
+    @classmethod
+    def clean_data(cls, df):
+        # We drop the 'label' column if it's available. For RIS files ASReview stores
+        # and loads the labels from the notes field.
+        df.drop("label", axis=1, inplace=True, errors="ignore")
+        return super().clean_data(df)
+
 
 class RISWriter:
     """RIS file writer."""


Test patch:
diff --git a/tests/test_readers.py b/tests/test_readers.py
index a36cadba..6cac50cd 100644
--- a/tests/test_readers.py
+++ b/tests/test_readers.py
@@ -7,6 +7,7 @@ from pytest import mark
 
 from asreview.data.loader import _from_file
 from asreview.data.loader import load_records
+from asreview.data.ris import RISReader
 from asreview.utils import _is_url
 
 
@@ -186,3 +187,29 @@ def test_load_records_from_url(tmpdir):
 def test_real_datasets(dataset_fp):
     data = load_records(dataset_fp)
     assert len(data) > 5
+
+
+def test_ris_lb_tag(tmpdir):
+    record = """TY  - JOUR
+LB  - 1
+TI  - This is the title
+ER  -
+
+TY  - JOUR
+LB  - 0
+TI  - This is the second title
+ER  -
+
+TY  - JOUR
+LB  - 28362428
+TI  - This is another title
+ER  -
+"""
+    fp = Path(tmpdir, "lb_data.ris")
+    with open(fp, "w") as f:
+        f.write(record)
+
+    records = RISReader.read_records(fp, dataset_id="foo")
+    assert len(records) == 3
+    for record in records:
+        assert record.included is None