SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We also aim to demonstrate the utility of our annotations by comparing them with the only other easily obtainable software annotation dataset, Softcite [32], a resource that provides annotations of software mentions in full text research publications in the life sciences and economics. Here, we partition FlaMBé's full text tool annotations into two sets of full text data, holding out 11 randomly chosen
Table 1: Predictive performance (P/R/F1 scores) of various language models. Language models were fine-tuned on a combination of full text and abstracts and evaluated on a mixture of both text types for cell type and tissue annotations. Best performers are highlighted in bold.
Cell Type	Tissue
Precision	Recall	F1	Precision	Recall	F1
BERT-base [39]	0.720	0.848	0.779	0.775	0.822	0.798
ELECTRA [40]	0.768	0.837	0.801	0.815	0.838	0.826
BioBERT [21]	0.740	0.847	0.790	0.776	0.869	0.820
BlueBERT [24]	0.706	0.872	0.781	0.790	0.865	0.826
BioELECTRA [23]	0.710	0.875	0.784	0.803	0.879	0.840
PubMedBERT [20]	0.737	0.858	0.793	0.830	0.857	0.843
Table 2: Predictive performance (F1 scores) of PubMedBERT on tool annotations when using Softcite or FlaMBé (excluding papers used for evaluation) as training standard. Tool annotations from 11 full text papers were held out from FlaMBé as an evaluation standard. PubMedBERT was fine-tuned on either the entirety of Softcite annotations or the smaller FlaMBé training standard.
Precision	Recall	F1
Softcite [32]	0.415	0.548	0.472
FlaMBé	0.779	0.887	0.830
papers for evaluation. We use the remaining 44 papers from FlaMBé and the entirety of Softcite for training. Both datasets were used to train PubMedBERT, one of the consistent performers in tissue/cell type prediction (Table 2). Despite being a smaller set of annotations, FlaMBé outperforms Softcite, especially when it comes to identifying the full name of a tool, (e.g., "Search Tool for the Retrieval of Interacting Genes/Proteins," more commonly known as "STRING"). This observation seems to be supported when we examine the predictive performance broken down by tag type—the largest performance difference between a model trained on Softcite and FlaMBé is in the 'I-Tool' token (see Supplement). We hypothesize that the fact that biomedical tools often have long, multi-word names (and corresponding acronym) may play in role in this large difference. Of course, we note that in this comparison FlaMBé has the advantage of using the same annotation criteria in both the training and test sets, but nevertheless, we believe it still illustrates the importance and utility of FlaMBé's biomedical specific tool annotations.
Use case 2: tool context prediction As a proof of concept, we also used FlaMBé's tool context annotations and trained a PubMedBERT model to predict a tool's context given the sentence in which it is mentioned, akin to sentiment classification. We assembled a small set of training (191 sentences over 28 papers) and test (45 sentences over 8 papers) data, limiting ourselves to sentences containing a mention of at least one of the top 5 most mentioned tools, Seurat, Cellranger, t-SNE, Monocle, and STAR, each of which can be applied in multiple contexts. We then trained PubMedBERT models to predict context for each sentences in a one vs rest framework, for contexts that are well represented in the test and training datasets: Alignment, Marker Genes, and Clustering. Each of the classifiers performed well, with the alignment  $(\mathrm{AUC} = 0.954)$  and marker gene  $(\mathrm{AUC} = 0.953)$  contexts being more distinguishable and clustering  $(\mathrm{AUC} = 0.810)$  being the most difficult. Given this promising performance on a test case, we anticipate that more sophisticated methods will be able to achieve consistently strong performance with our annotations.
Use case 3: visualization and exploration of different scientific workflows Different workflows can be extracted from FlaMBé's annotations, at different levels of specificity, either by highlighting the different tools used in a paper (Fig. 2A) or the different tool contexts in a paper (Fig. 2B). These can also be combined to extract more exact methodology (Fig. 3). Benchmarking papers or work introducing a new tool have to compare with previous work and create interesting workflows, as a small set of sample types is processed with slight variations through different levels of an entire pipeline depending on a paper's objective (Fig. 2). Meanwhile, papers that seek to solve a biological problem often have a more defined flow, with fewer tools from sample to one or more endpoints (Fig.
Figure 2: Sankey visualizations of (A) tool and (B) context workflows from an example paper [41]. Visualizations here focus on one entity at a time, either the computational tools being used throughout the paper (vertical bars in A) or the context (vertical bars in B) in which they are being used.
Figure 3: Sankey visualizations of the joint tool-context workflow from an example paper [42]. Visualization here depicts the workflow of (tool, context) pairs (vertical bars), where context is denoted within the parentheses.
3). By extracting these workflows, we can not only classify the type of paper (e.g., benchmarking, new method, or biological insight), and analyze them on an individual level, but can also look at the global set of workflows for a large set of papers (Fig. 1). Thus, FlaMBé has important downstream potential for extracting knowledge at multiple levels.
4.1 Potential downstream applications
There are many other interesting downstream applications that FlaMBé can be used to study. In addition to the advances in developing systematic methods for procedural knowledge extraction, we want to highlight the scientific value of improved modeling here. Specifically, structured representations would potentially allow for improved computational method recommendation depending on the goals of a particular study, as well as highlight gaps and areas of need for new computational method development. Importantly, one of the natural concerns that has been raised in psychology and more
recently in machine learning is that having ever more complex computational workflows can spawn multiverses. The multiverse represents the set of parallel universes where slightly different paths (e.g., methods or analysis steps) are taken towards the same goal. Multiverse analyses are undertaken to see how reliable results and conclusions are in light of these implicit decisions [43, 44]. We believe one of the most exciting downstream applications of FlaMBé is systematic multiverse analyses of the complex workflows undertaken in biomedical research, towards the ultimate goal of improving transparency and reproducibility of research claims.
One of the current limitations of FlaMBé is that though the number of entity-level annotations is high, there are relatively fewer examples of the more complex annotation types of tool context and workflow. We plan to address this through larger annotation efforts that will further expand these categories. Because FlaMBé has also proposed a systematic, structured representation that can be used as input to existing language models, these future efforts can be aided by computational predictions that can guide manual curation efforts. In these follow-up efforts, we foresee that the NER-related annotations will be easiest to automate, followed by NED, with the tool context and workflow predictions being more challenging. Any automated annotations will be reviewed by expert curators before release of an updated dataset. We do not foresee negative societal impacts, though incorrect workflows could potentially be misleading for downstream research, and thus we would encourage thorough evaluation of all predictions.
With FlaMBé, we have broken down the more complex, abstract procedural knowledge extraction problem into more structured declarative knowledge tasks that the community is already well-equipped to tackle. Intriguingly, cognitive psychology research has pointed towards the fact that in humans, procedural and declarative knowledge are intertwined, but can sometimes be learned independently of one another [45]. Thus, there may also be benefit to using different, more "procedural" representations for learning. In some sense, one ML area that has tried to learn and mimic human procedural knowledge is reinforcement learning. A good example of this is with "script knowledge" [46] and generally text-based games [47], which have used a game approach to improve modeling at the intersection of language understanding and complex decision-making. Reinforcement learning has also found some early success in reasoning over large scale knowledge graphs. Procedural knowledge extraction from academic texts could potentially also benefit from this type of framework. One of the unique aspects of FlaMBé is that though we have developed a structured representation, they can also tie together (e.g., we have annotated individual edges that can be viewed jointly as a graph). The disambiguated terms also tie in with existing knowledge bases that can be incorporated into knowledge graph research. It will be interesting to see whether new methods can be developed that could take advantage of the joint representation and learn more than the sum of the parts.
In conclusion, we have developed FlaMBé, a collection of datasets that together form structured representations of procedural knowledge captured in scientific literature. The dataset provides annotations for 1,195 paper abstracts and 55 full text papers, spanning over 700,000 tokens. In addition to providing the largest NER and NED dataset for tissue and cell type, we also provide annotations for computational tool and method, as well as the analysis task a tool is used in. Finally, we also annotate computational workflows within papers that can potentially be used in many downstream applications. Our dataset and associated code are accessible at https://github.com/ylaboratory/flambe.
This work was supported by the Cancer Prevention & Research Institute of Texas (CPRIT RR190065). VY is a CPRIT Scholar in Cancer Research.