Repository: acl-org/acl-anthology

Issue:
Compact bibfile
Overleaf has a 50 MB file size limit, and `anthology.bib` is now larger than this. We should create a compact BibTeX export using string substitution [as suggested here](https://twitter.com/daniel_hers/status/1744434842895294496). I'm not sure if this should just replace the current Anthology bib file, or become a new export, say `anthology-compact.bib`:

* The advantage of encompactifying the current file is it would work for everyone without having to change anything.
* The disadvantage is it complicates cutting-and-pasting.
* However, we already have [https://aclanthology.org/anthology+abstracts.bib.gz](https://aclanthology.org/anthology+abstracts.bib.gz). I'm not sure how often people cut-and-paste individual entries from the complete file, anyway; it seems that it's main use is in Overleaf.

I'm therefore include to simply replace `anthology.bib`.


Discussion:
I prepared a minified version which drops some fields that I don't think are important in bibliography, such as: _editor_, _month_, _address_, and duplicate _url + doi_. https://github.com/zouharvi/anthology-bib-small

Maybe this could be included in your minification so that instead of having the same issue in 2026, we have it in 2028, if LaTeX/Overleaf is still around by then?

That said, this is a very opinionated change to remove all these fields. However, except for the address, the output in ACL Natbib style would be the same, I believe.
Thanks for the effort and the initiative! As you suspected, though, we can't officially sanction this because it removes information that is important for complete and correct citations. This issue comes up, for example, in tenure and promotion considerations. We recently added the editor field (#2706) which probably contributes to the increased file size.

The approach I think we should take would be to use the string substitutions [suggested here](https://danielhers.github.io/mrp.bib).
Another possible solution is to split the ever-growing `anthology.bib` into multiple 50MB-each files and have permlink `anthology-1.bib`, `anthology-2.bib` and so on.
It would be quick to implement, but I dislike the idea of producing multiple file when the simplicity of a single one is in reach. It would also be hard to undo this once people became dependent on it.

I wonder if i could interest anyway in rolling up their sleeves and implementing the string-based approach suggested above. Here is a sketch of what it would look like:

- The script is [https://github.com/acl-org/acl-anthology/blob/master/bin/create_bibtex.py](create_bibtex.py). You can invoke it as follows: `python3 bin/create_bibtex.py --clean`. This will create a file `build/` filled with BibTeX files for every paper in the Anthology, as well as the consolidated variants, including `anthology.bib`.
- A simple trick that would probably be sufficient would be to create a `@string` for every volume, since every paper in a volume has the same booktitle. What's easy about this is that each volume ID is composed of an ASCII venue ID, a four-digit year, and a volume name, which are all permissible characters.

For example, 

```bibtex
@proceedings{yrrsds-2023-young,
    title = "Proceedings of the 19th Annual Meeting of the Young Reseachers' Roundtable on Spoken Dialogue Systems",
    editor = "Hudecek, Vojtech  and
      Schmidtova, Patricia  and
      Dinkar, Tanvi  and
      Chiyah-Garcia, Javier  and
      Sieinska, Weronika",
    month = sep,
    year = "2023",
    address = "Prague, Czechia",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.yrrsds-1.0",
}
```

will become

```bibtex
@string{YRRSDS:2023:1 = {Proceedings of the 19th Annual Meeting of the Young Reseachers' Roundtable on Spoken Dialogue Systems}}
@proceedings{yrrsds-2023-young,
    title = YRRSDS:2023:1,
    editor = "Hudecek, Vojtech  and
      Schmidtova, Patricia  and
      Dinkar, Tanvi  and
      Chiyah-Garcia, Javier  and
      Sieinska, Weronika",
    month = sep,
    year = "2023",
    address = "Prague, Czechia",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.yrrsds-1.0",
}
```


Resolving patch:
diff --git a/bin/create_bibtex.py b/bin/create_bibtex.py
--- a/bin/create_bibtex.py
+++ b/bin/create_bibtex.py
@@ -27,6 +27,7 @@
   -h, --help               Display this helpful text.
 """
 
+import re
 from docopt import docopt
 from tqdm import tqdm
 import gzip
@@ -70,9 +71,20 @@ def create_bibtex(anthology, trgdir, limit=0, clean=False) -> None:
     ) as file_anthology, gzip.open(
         "{}/anthology+abstracts.bib.gz".format(trgdir), "wt", encoding="utf-8"
     ) as file_anthology_with_abstracts:
+        # Add some shortcuts to the consolidated bib file
+        print(
+            "@string{acl = {Association for Computational Linguistics}}",
+            file=file_anthology_raw,
+        )
+        print("@string{anth = {https://aclanthology.org/}}", file=file_anthology_raw)
+        print(file=file_anthology_raw)
+
         for volume_id, volume in tqdm(
             sorted(anthology.volumes.items(), key=volume_sorter, reverse=True)
         ):
+            # reset this each time
+            abbrev = None
+
             volume_dir = trgdir
             if not os.path.exists(volume_dir):
                 os.makedirs(volume_dir)
@@ -91,6 +103,53 @@ def create_bibtex(anthology, trgdir, limit=0, clean=False) -> None:
                         concise_contents = paper.as_bibtex(concise=True)
                         print(concise_contents, file=file_volume)
                         print(concise_contents, file=file_anthology)
+
+                        # Space saver (https://github.com/acl-org/acl-anthology/issues/3016)
+                        # Replace verbose text with abbreviations to get the file under 50 MB for Overleaf
+
+                        concise_contents = concise_contents.replace(
+                            'publisher = "Association for Computational Linguistics",',
+                            "publisher = acl,",
+                        )
+                        concise_contents = re.sub(
+                            r'url = "https://aclanthology.org/(.*)"',
+                            r"url = anth # {\1}",
+                            concise_contents,
+                        )
+
+                        # Abbreviate the booktitle by extracting it and printing it before
+                        # the first entry in each volume
+                        if concise_contents.startswith("@proceedings"):
+                            # Grab the title string and create the alias
+                            abbrev = f"{volume.get_venues()[0].upper()}:{infer_year(volume.collection_id)}:{volume.volume_id}"
+                            try:
+                                booktitle = re.search(
+                                    r"    title = \"(.*)\",", concise_contents
+                                ).group(1)
+                                print(
+                                    f"@string{{{abbrev} = {{{booktitle}}}}}",
+                                    file=file_anthology_raw,
+                                )
+                            except AttributeError:
+                                import sys
+
+                                print(
+                                    f"Could not find title for {volume_id}",
+                                    file=sys.stderr,
+                                )
+                                abbrev = None
+
+                        if abbrev is not None and "booktitle" in concise_contents:
+                            # substitute the alias for the booktitle
+                            concise_contents = re.sub(
+                                r"    booktitle = (\".*\"),",
+                                f"    booktitle = {abbrev},",
+                                concise_contents,
+                            )
+
+                        # Remove newlines, indentations, and double-spaces around author separators
+                        concise_contents = re.sub(r"\s+", " ", concise_contents)
+
                         print(concise_contents, file=file_anthology_raw)
 
 


Test patch: