dataesr/scientific-paragraphs-categorization huggingface dataset:
14 ± 0. 013, with very little variation across individuals. In contrast, when using the sequence similarity metric, the average per-individual fraction was substantially higher (0. 22 ± 0. 087), with much greater variability across individuals. Thus, while per-individual repertoires seem to differ substantially when only sequence similarity is concerned, these repertoires are revealed to be much more similar in their structure/function coverage when represented by AbMAP. We wondered if the distribution pattern of human antibody representations is therapeutically useful. Towards that, we mapped 547 antibodies from Thera-SABDab, a dataset of immunotherapeutic antibodies that have entered clinical trials  (40). Our hypothesis was that even  if an antibody drug candidate is effective in vitro, it may not have drug-like properties (e.
Predicting SARS-CoV-2 Variant Neutralization Ability We also examined AbMAP's capability on the prediction of antibody efficacy on neutralizing SARS-CoV-2 variants. This analysis also demonstrates how AbMAP could be applied in a low-N setting to optimize an existing panel of antibodies to address incremental mutations in the target. From CoV-AbDab, a dataset of coronavirus-binding antibodies, we obtained the set of 2,077 antibodies that were reported to neutralize the wild-type strain of SARS-CoV-2; we then computed fixed-length AbMAP-B embeddings for these. Setting aside 522 (approx 25%) randomly chosen antibodies, we plotted the rest (=1,555) on the 2D visualization described above (Fig. 5a ). The set of wild-type neutralizing antibodies span a fairly large part of the overall space, suggesting that a substantial diversity of antibody structures is capable of neutralizing SARS-CoV-2 (we note that these antibodies vary in their reported viral target protein).
Discussion We presented AbMAP, a transfer learning framework to adapt any foundational PLM (i. e. , one trained on the broad corpus of protein sequences) to antibodies, whose hypervariable regions violate the evolutionary conservation that PLMs typically rely on. Our framework refines the raw PLM embedding by in silico mutation-based contrastive augmentation. To further shape the augmented embedding, we supervise its non-linear projection to a lower dimensional space such that Euclidean distances in the projected space better capture antibody structural (from PDB) and functional (from Setliff et al. 's LIBRA-seq (  23 )) similarity. In an ablation study, we found that training on just structural data recapitulated function but not vice-versa (Table  5 ). This may be due to single-cell data sparsity and noise in the LIBRA-seq dataset.
Correspondingly, our multi-task formulation also learned a higher weight for the structural task. To maximally leverage the limited structural data on antibodies, AbMAP focuses its capacity on the complementarity-determining regions (CDRs) and their flanking residues, these being the key drivers of antibody specificity. We present AbMAP-E, AbMAP-P, and AbMAP-B, adaptations of the ESM-1b, ProtBert, and the Bepler & Berger PLMs, respectively. The choice of foundational PLM may depend on the type of downstream prediction task using the resulting AbMAP embedding (e. g. , we found AbMAP-B to be better for structure prediction, and AbMAP-E/P for function/property prediction). Our transfer learning framework can be adapted to any new PLM, and we make both the training data and code available for doing so.
AbMAP represents a conceptual advance to language model design for antibodies. Currently, two broad approaches have been espoused. One, embodied in techniques like OmegaFold  (26) , is to essentially focus on improving the general-purpose PLM, with the expectation that gains will accrue to antibodies as well. The other, represented by methods such as AntiB-ERTa (13), AbLang  (42) , and IgLM (  14 ), is to essentially treat antibodies as an entirely new language-modeling task and train solely on corpora of antibody sequences. AbMAP represents a new middle path. We describe a transfer learning approach that can adapt to any foundational PLM and hence benefit from innovations in the underlying PLMs. Since antibody hypervariable regions are not evolutionarily conserved, foundational PLMs will likely remain weaker at modeling antibodies than regular proteins.
On the other hand, the framework regions (which comprise ∼90% of the sequence) have limited diversity across antibodies. Language models focused specifically on antibodies thus do not benefit from the full diversity of protein sequences that foundational PLMs access. Our approach starts from informative foundational PLMs and then adapts them to be more accurate on antibody structure and function. We additionally make the choice to focus AbMAP's explanatory capacity on the CDRs and their flanking residues. This is an inductive bias, allowing us to limit the model complexity of the transfer learning layer (one layer of transformer) and enabling robustness as well as accuracy. Lastly, an intriguing direction of future work would be to combine ideas from AbMAP's transfer learning approach with repertoire-based PLMs (like AntiBERTa) to infuse structural, functional as well as repertoire information into antibody representations.
Thus, our CDR-focused approach more representatively captures the diversity of antigen-binding profiles in an individual repertoire. Nonetheless, in some cases of thera-peutic antibody design, the framework region may play an important role. To address this, we followed Liberis et al. (18)  and expanded the standard Chothia delineation of a CDR to also include two flanking residues on each end. In future, as more antibody datasets become available (e. g. , those measuring binding or functional specificity in diverse contexts), the additional training data may enable us to leverage more complex models that effectively capture both the framework and hypervariable regions. The exploration of human immune repertoires, as well as the design and development of large-molecule therapeutics, require a deep understanding of antibody structure and function, and an ability to efficiently manipulate it in silico.
However, the hypervariability of antibodies poses a challenge for general-purpose protein modeling techniques, hindering their translation to antibodies. The transfer learning approach of AbMAP represents a broadly-applicable solution to such conundrums. Rather than training a dedicated language model from scratch for a sub-category of proteins, we argue it is more effective to leverage the fast-moving advances in foundational PLMs and fine-tune a general PLM towards the subset. AbMAP's transfer learning framework enables us to effectively adapt foundational PLMs. We believe the scalable and accurate modeling of antibodies it enables will unlock a better understanding of the behavior of antibodies, empowering the discovery of novel therapeutic biologics. For each antibody, this set of three scores (each standardized to µ = 0, σ = 1) serves as its binding specificity vector.
CoV-AbDab CoV-AbDab is a database of all published/patented antibody sequences (7,964 sequences in total as of July 19, 2022) that are capable of binding to coronaviruses including SARS-CoV-2, SARS-CoV-1, and MERS-CoV  (48). To evaluate our trained model, we selected from these the 2,077 sequences that bind to wild-type SARS-CoV-2. On these, we assessed the model's ability to predict if the antibody could also neutralize at least one SARS-CoV-2 variant (alpha, beta, omicron, etc. ), using only the antibody sequence information.
Datasets for Labeled Antibody Mutants The dataset provided in Desautels et al. (  2020 ) was used to validate our antibody language model through ddG score predictions  (31). They used a machine learning model to search the mutational combinatorial space of the m396 antibody, and calculated the binding propensity of these mutants against the SARS-CoV-2 spike protein's receptor binding domain. They applied five standard software packages to score the physicochemical characteristics of the mutant's binding, including STATIUM, FoldX and Rosetta  (30, 28, 29). To validate our model, we used these pre-computed scores to formulate regression-based property prediction tasks, with only the mutant sequence as input.
Embedding Generation via mutational augmentation Given a heavy or a light chain sequence of length n of an antibody, we created its CDR-specific embedding by performing the following set of operations. We first create an R n×d embedding using an foundational protein language model such as the Bepler & Berger (9), ESM-1b (8), or ProtBert  (10) , where d = 2200 (Bepler & Berger), 1280 (ESM-1b), or 1024 (ProtBert). We note that Bepler & Berger has 6165 dimensions across three layers but we used only the last layer's weights. Then, using ANARCI (20), we number the amino acid residues in the input sequence using the Chothia scheme  (21). Then, we identify parts of the embedding that correspond to the CDRs based on the numbering scheme. Next, we generate k (here, k = 100) new antibody sequences through in silico mutagenesis of the original sequence, where the mutation is performed (sampling from a uniform distribution over all amino acids) with certain probability (here, 0.
Language Model Refinement with a Multi-task Architecture Once we generate a CDR-specific embedding for a given antibody sequence, we use it as input to our model which outputs a fixed-length feature that can be used for further downstream taskspecific similarity prediction. We curated two datasets for pairs of antibodies, one labeled with their 3D structure similarity, and the other with functional similarity (with regards to antigen binding profiles) to train the model. The overall pipeline as well as the diagram of our antibody language model is shown in Fig. 1. Our model consists of an MLP projection module whose parameters are shared across the training of each task (structure and function). This projection module reduces the dimension of the input embedding (R n ×d → R n ×d , d < d) so that the language model is forced to retain as much information about the antibody sequence as possible while trying to execute downstream prediction tasks.
For example, when using the Bepler & Berger language model as the foundational PLM, the input CDR-specific embedding is R n ×2200 and our model's projection module outputs a R n ×256 embedding. Then, the embedding with reduced dimensions, outputted by the projection module is fed into two separate PyTorch (v1. 11. 0) Transformer Encoder modules, one each for a downstream similarity score prediction task (structure and function)  (49). We add sinusoidal positional encodings to the input embeddings in order to inject information about the relative positions of amino acid residues in each sequence. The representations for each residue in the embedding outputted by this Transformer Encoder module now incorporate antibody-related structural and functional information by leveraging attention from other residues in the CDR.
Datasets SAbDab SAbDab (Structural Antibody Database)  (22)  is a database of antibody structures where each structure is paired with various metadata including the heavy and light chains of antibodies in the PDB complex. Limiting ourselves to PDB entries where both the heavy and light chains were available, we obtained 3,785 pairs of heavy and light chain antibody sequences and PDB structures from SAbDab. We divided them into 3,000 pairs for train and 785 pairs for evaluation. From the 3,000 antibodies selected for training, we randomly sampled 100,000 pairs of antibodies and computed the pairwise similarity scores using their PDB structural data. For each of the sampled 100,000 SAbDab antibody pairs, we computed the TM-Scores of the pairs of antibodies (separately for heavy and light chains), and used them as the ground-truth label when supervised against the predicted similarity score from our model.
Max-Entropy regularization In order to prevent the learned representations being overfit to the datasets used for supervision, we include a regularization loss adapted from Shannon entropy (50): The regularization is applied to the antibody's L 2 -normalized fixed-length feature of length q, which is outputted by the task-specific Transformer Encoder module following the nonlinear projection module. The square of each entry in the fixed-length feature is treated as the probability of an arbitrary discrete random variable X, corresponding to P (x i ) in the above equation. To induce a regularizing effect, we want the squared entries in the feature to form a uniform distribution. For each L 2 -normalized feature, the squared entries are non-negative and sum to 1, like a probability distribution.
The only exception is s4to4+lrn+pers+PSE (last line), which gains more than two accuracy points over baseline. However, this result is solely driven by the net improvement on deixis disambiguation (almost +5 points, see Table  10 ), while the performance is degraded on the other three discourse phenomena. In conclusion, sentence position encodings do not seem to benefit the vanilla s4to4 approach. With the aim of evaluating the generalizability of these results to another language pair and domain, we train the context-discounted approach on the En→De IWSLT17 dataset and evaluate it on ContraPro  (Müller et al. , 2018). 5  Table  3  summarizes the results. Unfortunately, the improvements achieved on En→Ru do not transfer to this setting. The s4to4+CD slightly benefits from segment-shifted position embeddings, but the other approaches degrade its performance.
Conclusions Intending to improve concatenation approaches to context-aware NMT (CANMT), we investigated an intuitive idea: encoding into token representations the position of their sentence within the processed sequence. Besides adopting existing encoding methods (segment-shifted position embeddings and segment embeddings), we proposed a novel approach to integrate token and sentence position embeddings in a unique vector called position-segment embedding (PSE). We also propose to make sentence position encodings persistent throughout the model's layers. We compared these encoding approaches on the En→Ru/De language pairs. Consistent improvements were observed on En→Ru when persistent sentence position encoding methods were used in conjunction with the context-discounted training objective proposed by  Lupo et al.
C Increasing training data for the English to German pair We hypothesize that the model does not undergo sufficient training in the En→De setting to reap the benefits of segment embeddings. Indeed, the training data volume is smaller than in the En→Ru setting: 0. 2 million sentences versus 6 million (see Table  6 ). Therefore, we choose to experiment with more En→De training data, employing the same high-resource setting of  Lupo et al. (2022a). This setting expands the IWSLT17 training data  (Cettolo et al. , 2012).
Hosting a Pedagogical engineering with the support of the UNFM: This training management platform is based on ve tools: (1) A Learning System to create courses, based on a TCC (Training, Coaching, Communities) process We used a TCC process de ned as: Training: all opportunities will be offered to learners as well as teachers on the principle of empowerment. Coaching: interaction between learners and trainers in synchronous or asynchronous mode (video, audio, dialogues, academic exchanges). Communities: health professionals identi ed and a liated to a community of practice network of experts in oncofertility and oncosexuality are involved in this website. The sharing of skills, initiatives and collaborative work practices will be privileged and encouraged on this platform (2) A Web Hosting with pedagogical engineering including appropriate applications for online authoring tools to produce courses, tests and multimedia content The innovation is technological (broadcasting on low-speed internet, free interactive software DUDAL) and organizational (pedagogical moderating resources).
A responsive design: Through innovative tools, the digitization of training will allow the provision of courses in the form of modules. The educational resources will be adapted for quality access, even at reduced Internet speeds. They will be able to bene t from a system that will synchronize the recording, the multilingual subtitling of the audio streams, and the live compression of the courses with speci c software, DUDAL, from any Internet connection. DUDAL is the ideal tool used to put the courses online. The video conferences are produced with this free software developed for RAFT (Network in Francophone Africa for Telemedicine) to work with low bandwidth connections (25 kb/s) (12). It allows participating in educational sessions either as a listener or speaker, with the possibility of an interactive forum.
As well, we propose (i) a thematic database on oncofertility and oncosexuality, (ii) a standardized and widely used framework for evaluating e-learning programs using the UNFM, and (iii) structured programs that integrate e-learning between recognized scienti c institutions (CECOS, GEFRAUS, Toulouse University). New horizons will open up for our scientists and physicians regarding training and learning aligned with an innovative digital strategy. These innovations are necessary to overcome the di culties of coordination and training between local professionals. Related work will contribute to a continuum for implementing an ad-hoc training library. In addition, the innovators' a liation centers will be identi ed in the Caribbean as centers of competence and excellence in sexual and reproductive health care with high standards of clinical practice.
This program can be replicated in other world regions, thanks to its lowcost interactivity and UNFM's global expertise in deploying training in developing countries. In this dynamic setting, the health care system will also have to evolve, if not revolutionize, in terms of organization and management. Content distribution and targeting are levers for optimizing site tra c and generating quali ed leads. Digitalization of audio-visual pedagogical resources combined with the performances of the networks provided by the RAFT offer many possibilities. Indexing such resources is a major challenge for expert practice community. Furthermore, the provision of high-tech equipment and the expertise of the UNFM will generate a network of community exchanges for continuing professional development and reinforcement of health behaviors as close as possible to the local health territories.
Acknowledgements The authors thank all the other contributors from the  University Hospitals of Martinique, Guadeloupe and Toulouse ; the  Ministry of Public Health of Cuba , the  Cancer Registries of Cuba  and  Martinique ; the  National Institute of Oncology and Radiobiology of Cuba , the UNFM; the  French Embassy in Cuba  and the  French and Cuban National Authorities , notably the following persons for their valuable insights and support during the project development: Pr. Carlos Alberto MARTINEZ BLANCO, Oncology referent at MINSAP in  Cuba Pr. Roberto ÁLVAREZ FUMERO , Director of the Maternity and  Childhood Department  at the  Ministry of Public Health of Cuba. Pr. Rodolfo ENRIQUEZ RODRIGUEZ, Director General of Ramón González Coro Hospital, Cuba. Dr. Miguel Hernán Estévez DEL TORO , Director General of  Hermanos Ameijeiras Hospital, Cuba.
This phenomenon is very clear and popular today, with the emergence of machine learning in deep neural nets with big data and large language models and consequently the emergence of interactive, generative artificial intelligence systems such as ChatGPT. As colleagues at the University of Edinburgh (https://www. ed. ac. uk/informatics/about/what-is-informatics) have synthesized the subject, informatics, the science of information, consists of computer science (the components of modern computers, including robots), cognitive science (how animals, individually and collectively, perceive, understand, reason, decide and act) and artificial intelligence, which links the two complementary disciplines. This vision of informatics includes 1. Technical/mathematical components, such as logic, complexity theory, algorithms, software, databases, hardware, and networks typical of computer science; the anatomy of artificial systems.
the strong dependence of these on the context. Both concerns remain at the core of modern informatics, thus encouraging a joint project. Prof. Krief's vision of the future of chemistry was one of a pioneer, without any doubt. A crucial point concerns the difference between "information" and "knowledge. " We were perfectly aware that it would be possible to build a repository of information, a kind of electronic library of concepts, properties of concepts, and relations among concepts and their properties, often called "ontologies. " There are many databases, particularly in chemistry; however, the goal of EnCOrE was much more ambitious. Prof. Krief explained that he wanted to identify and implement a process of construction and use of chemical knowledge, not merely chemical information. For years, we discussed the difference between information and knowledge.
Both these critical technical aspects were discussed with Prof. Krief, whose enthusiasm and engagement were unlimited. [6]  ). Below are a few snapshots of the work undertaken with Prof. Krief, fully described in the papers and deliverables for the EU. The Grid Shared Desktop was the interface of a system called AGORA, which enabled the definition and exploitation of Agents (human and eventually software agents), Groups (sets of Agents), Organizations (the structure of rights and duties of groups), Resources (computational and communication resources), and activities (the processes activated within the system). AGORA interactions run on Grid middleware (i. e. , worldwide on the web) services (processes activated by software) were stateful, i. e. , with memory, enabling the storage and retrieval of histories of the interactions.
The session involved the use of different shared applications such as Flashmeeting (http://flashmeeting. open. ac. uk/) for communication (courtesy of the Open University, Knowledge Media Institute, UK: Prof. Marc Eisenstadt, one of our partners), ChemDraw (a standard software for drawing chemical elements), Resyn Assistant (a The duties and rights of the five components of AGORA, the core software behind its interface, the Grid Shared Desktop, were set previously to coordinate the reasoned exploitation of the collaborative processes during the interactions at a distance. To the author, no other software tool currently offers the same robust control and communication functionalities. Human learning in these experiments occurred in many ways because of the interactions. Each participant "learned" continuously from the others and the whole experience.
Two examples highlight how this experiment was an exceptional source of human learning at a distance. The first was linked to Prof. Krief, a senior scientist learning to use complex software. Previously, he had never been in a situation where he could build an ontology, i. e. , a written, formal symbolic representation of concepts, properties, and relations, that software could use to infer new concepts/properties/relations. His profound knowledge of chemistry was indispensable for the success of the enterprise. No other (younger) scientist -probably better acquainted with the tool -could replace him. Therefore, he needed to  "understand and exploit" the powerful, complex Protégé tool. Monica Crubezy, the author of the interface, helped him (at a distance) to become familiar with Protégé in the context of knowledge construction.