Repository: asreview/asreview

Issue:
RIS files with A1 (and A2, A3, A4?) tags are corrupted on the export

Current implementation of RIS exporter in ASReview doesn't handle author tags in RIS incorrectly (if not `AU`). RIS supports 5 tags for authors: `AU`, `A1`, `A2`, `A3`, `A4`. Currently we only support AU, see: 

https://github.com/asreview/asreview/blob/2572dd425c093e7c193effbf1dcfcd113c33e72f/asreview/io/ris_writer.py#L51

## Relevant code

I'm not sure what the actual issue is at the moment. Some relevant code snippets are:

```python
>>> from rispy import TAG_KEY_MAPPING
>>> from pprint import pprint
>>> pprint(TAG_KEY_MAPPING)
{'A1': 'first_authors',
 'A2': 'secondary_authors',
 'A3': 'tertiary_authors',
 'A4': 'subsidiary_authors',
 'AB': 'abstract',
 'AD': 'author_address',
 'AN': 'accession_number',
 'AU': 'authors',
```

ASReview only recognizes `AU` and `A1`:
https://github.com/asreview/asreview/blob/2572dd425c093e7c193effbf1dcfcd113c33e72f/asreview/config.py#L49


## Example of export from Mendeley 

```
TY  - JOUR
T1  - The effect of offenders' sex on reporting crimes to the police
A1  - Wong, Thessa M.L.
A1  - van de Schoot, Rens
Y1  - 2012/05//
KW  - Crime reporting
KW  - Robbery with assault
KW  - Sex differences
KW  - Victim-offender relationship
KW  - Victims
JF  - Journal of Interpersonal Violence
VL  - 27
IS  - 7
SP  - 1276
EP  - 1292
DO  - 10.1177/0886260511425245
N2  - This article examines the difference in victims' reporting behavior regarding crimes committed by males and by females. The authors expect that victims of female offenders are less likely to report to the police than victims of male offenders because of differences in the victim-offender relationship as well as in the victim's sex. With recent developments in Bayesian statistics, new tools have become available that enable the direct evaluation of researchers' expectations. All cases of robbery with assault from the National Crime Victimization Survey have been investigated (n = 478). Findings reveal that female offenders are underreported compared with male offenders and that this can be explained by the victim characteristics but only in combination with the offender's sex. © The Author(s) 2012.
ER  - 
TY  - JOUR
T1  - Meaning-in-life orientations and values in youth: Cross-cultural comparison
A1  - Gorlova, Natalia
A1  - Romanyuk, Lyudmyla
A1  - Vanbrabant, Leonard
A1  - van de Schoot, Rens
Y1  - 2012/11//
KW  - Meaning of life
KW  - Measurement invariance
KW  - Purpose in life
KW  - The Meaning-in-Life Orientations test
JF  - European Journal of Developmental Psychology
VL  - 9
IS  - 6
SP  - 744
EP  - 750
DO  - 10.1080/17405629.2012.689823
N2  - Search for purposes in life and the meaning of life is one of the developmental tasks in youth. The Meaning-in-Life Orientations test (MOL) is often used to assess purposes in life and meaningfulness of life and is often used to compare individuals and groups. It is yet unclear whether this test shows measurement invariance across different countries or not, which was the main object of the current study. First, confirmatory factor analyses were conducted separately for data sets with Russian and Ukrainian students. Second, metric and scalar invariance were tested. Finally, comparisons between Russian and Ukrainian students were made on their scores on the MOL. © 2012 Copyright Taylor and Francis Group, LLC.
ER  - 
```


![image](https://user-images.githubusercontent.com/12981139/157640876-7bd28d35-3b14-4eaa-9bd1-334ac8c54f3f.png)


## Quickfix / temp fix

1. Open the RIS file in a text editor
2. Replace `A1  - ` by `AU  - `
3. Open file in ASReview

Discussion:
@J535D165: Command line fix:

```
sed 's/A1  - /AU  - /' data/NAME_OF_DATA.ris
```

Use `-i` to replace inplace.@J535D165: How to fix a corrupted output:

```python
import rispy

# load data

ris_lines = []
with open('data/my_file.ris') as bibliography_file:

    lines = bibliography_file.read().splitlines()

    for line in lines:
        if line != "A1  -  ":
            ris_lines.append(line)
        else:
            ris_lines.append("A1  -  <RRRRRR>\n")

bibliography = rispy.loads("\n".join(ris_lines))


# fix data

fixed_records = []

for record in bibliography:

    new_record = record.copy()
    new_record["authors"] = eval("".join(record.get("first_authors", ["[]"])).replace("<RRRRRR>", " "))

    try:
        del new_record["first_authors"]
    except KeyError:
        pass
    fixed_records.append(new_record)

print(fixed_records[0])

# export data

with open('data/my_file_fixed.ris', 'w') as fixed_file:
    rispy.dump(fixed_records, fixed_file)

```@PeterLombaers: I tried to reproduce this. I imported the example above into ASReview (the current main branch) and then exported it again. The RIS file looked fine. I did not try loading it into Mendeley.@J535D165: Thanks for checking! Will close the issue

Solution patch:
diff --git a/asreview/data/config.py b/asreview/data/config.py
deleted file mode 100644
index 1230b5fb..00000000
--- a/asreview/data/config.py
+++ /dev/null
@@ -1,39 +0,0 @@
-# Copyright 2019-2025 The ASReview Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-__all__ = [
-    "COLUMN_DEFINITIONS",
-]
-
-"""
-
-"""
-COLUMN_DEFINITIONS = {
-    "included": [
-        "final_included",
-        "label",
-        "label_included",
-        "included_label",
-        "included_final",
-        "included",
-        "included_flag",
-        "include",
-    ],
-    "title": ["title", "primary_title"],
-    "authors": ["authors", "author names", "first_authors"],
-    "abstract": ["abstract", "abstract note", "notes_abstract"],
-    "notes": ["notes"],
-    "keywords": ["keywords"],
-    "doi": ["doi"],
-}
diff --git a/asreview/data/ris.py b/asreview/data/ris.py
index dd22a8e1..307f6cea 100644
--- a/asreview/data/ris.py
+++ b/asreview/data/ris.py
@@ -23,6 +23,7 @@ import pandas as pd
 import rispy
 
 from asreview.data.base import BaseReader
+from asreview.data.utils import convert_to_list
 from asreview.utils import _is_url
 
 ASREVIEW_PARSE_RE = r"\bASReview_\w+\b"
@@ -215,6 +216,19 @@ class RISReader(BaseReader):
         # We drop the 'label' column if it's available. For RIS files ASReview stores
         # and loads the labels from the notes field.
         df.drop("label", axis=1, inplace=True, errors="ignore")
+
+        # We combine the values of 'authors' and 'first_authors' into one list of
+        # authors. Internally we only use the authors when searching for a record, so
+        # it does not matter too much if combining the two lists leads to the wrong
+        # order of authors, or duplicate authors appearing in the list.
+        if "authors" not in df:
+            df["authors"] = None
+        df["authors"] = df["authors"].apply(convert_to_list)
+        if "first_authors" not in df:
+            df["first_authors"] = None
+        df["first_authors"] = df["first_authors"].apply(convert_to_list)
+        df["authors"] = df["authors"] + df["first_authors"]
+
         return super().clean_data(df)
 
 
diff --git a/asreview/data/tabular.py b/asreview/data/tabular.py
index b2f313ef..230c3c45 100755
--- a/asreview/data/tabular.py
+++ b/asreview/data/tabular.py
@@ -18,7 +18,7 @@ __all__ = ["CSVReader"]
 import pandas as pd
 
 from asreview.data.base import BaseReader
-from asreview.data.config import COLUMN_DEFINITIONS
+from asreview.data.record import Record
 
 
 class CSVReader(BaseReader):
@@ -106,9 +106,9 @@ class ExcelReader(BaseReader):
 
         best_sheet = None
         sheet_obj_val = -1
-        wanted_columns = []
-        for _, type_list in COLUMN_DEFINITIONS.items():
-            wanted_columns.extend(type_list)
+        wanted_columns = Record.get_columns()
+        for col_names in cls.__alternative_column_names__.values():
+            wanted_columns.extend(col_names)
 
         for sheet_name in dfs:
             col_names = set([col.lower() for col in list(dfs[sheet_name])])


Test patch:
diff --git a/tests/test_readers.py b/tests/test_readers.py
index 6cac50cd..a6125ecf 100644
--- a/tests/test_readers.py
+++ b/tests/test_readers.py
@@ -213,3 +213,34 @@ ER  -
     assert len(records) == 3
     for record in records:
         assert record.included is None
+
+
+def test_ris_author_tags(tmpdir):
+    record = """TY  - JOUR
+AU  - author0
+AU  - author1
+AU  - author2
+TI  - This is the title
+ER  -
+
+TY  - JOUR
+A1  - author0
+A1  - author1
+A1  - author2
+TI  - This is the second title
+ER  -
+
+TY  - JOUR
+AU  - author0
+A1  - author1
+A1  - author2
+TI  - This is another title
+ER  -
+"""
+    fp = Path(tmpdir, "author_tags_data.ris")
+    with open(fp, "w") as f:
+        f.write(record)
+
+    records = RISReader.read_records(fp, dataset_id="foo")
+    for record in records:
+        assert record.authors == ["author0", "author1", "author2"]