dataesr/scientific-paragraphs-categorization huggingface dataset:
Even human annotators have difficulties in interpreting a meme correctly without knowledge of the community in which the meme was shared. In this paper, we tackle the shared task on multimodal semantic role labeling of the CON-STRAINT'22 workshop  (Sharma et al. , 2022). Given a (meme, entity) pair, 1 the goal is to classify the entity's role in the meme into one of four classes (hero, villain, victim or other) from the perspective of the author of the meme. The multimodality of the problem stems from the meme, which is given as an (image, OCR) pair, where OCR (for Optical Character Recognition) is the caption extracted from the image. The dataset covers one language, English, and two domains, COVID-19 and US politics. Figure  1  shows a sample from the training set. Understanding memes involves a lot of commonsense and cultural knowledge on the political stance of the entities.
Thus, it requires models pre-trained on a large amount of data, capable of recognising key entities such as political figures in both modalities, and of inferring their relationship, their role and the public opinion of a community on them. To evaluate the task's difficulty, we manually annotate a set of samples. With 5 annotators, we reach an average Macro-F 1 of 0. 65 (see details in Appendix A), less than 10 points above the best system submitted to the shared task. We propose systems relying on several multimodal (vision-language) pre-trained models: One For All (OFA,  Wang et al. , 2022) , CLIP  (Radford et al. , 2021)  and  VisualBERT (Li et al. , 2019). We use these models as encoders to extract multimodal meme representations. These encoders are introduced in Section 3. We then design several neural network classifiers to handle these representations in a task-specific fashion.
These classifiers are presented in Section 4. 1. The CONSTRAINT'22 dataset is characterised by a large class imbalance, with the most frequent class gathering 78% of the samples in the train set, while the least frequent one is conveyed by less than 3% of the samples. However, the challenge is evaluated using a Macro-F 1 metric and calls for balanced performances across all classes. To handle this discrepancy, we developed several subples, thus considering all entities of a meme independently during training and inference. sampling strategies that we present in Section 4. 2. Our best results are obtained by ensembling predictions from all of our models, using various ensembling methods. The details of the ensembling methods are given in Section 4. 3. Finally, we present our performance in Section 5 along with a qualitative analysis of our models.
We highlight the limitations of the dataset, task and methods in Section 6. To summarise, our whole architecture is built on freely available pre-trained models. We only fine-tune these models for the multimodal semantic role labeling task. This makes computational training cost particularly low. Our system can be characterised by: • Simple classifier design on top of deep pretrained model. • Handling of class imbalance through carefully-designed sampling strategies. Our code is available at: https://github. com/smontariol/mmsrl_constraint.
This involves completely different features compared to labeling the thematic relations of the entity; in particular, cultural and contextual knowledge on the background of the meme. Another similar task is multimodal named entity recognition, which aims at identifying and classifying named entities in texts and images. It requires more in-domain knowledge compared to multimodal SRL; but most multimodal NER datasets are text-centric, with the image being an additional feature for the text-based prediction  (Arshad et al. , 2019; Chen et al. , 2021) , while our task is more symmetrical or even image-centric. Finally, many shared task on memes have been proposed in recent years, with a large variety of tasks: emotion classification (e. g. MEMOTION task at SemEval 2020  Sharma et al. , 2020) ; hateful meme detection (e.
Multimodal Encoding Since we experiment with deep neural networks, we need to obtain distributed representations of our inputs. To this end, we use pre-trained mod-els with good performances on popular datasets. These models are multimodal transformers, that we use to encode image and caption's OCR into a common latent space. While transformers were originally developed for natural language processing  (Vaswani et al. , 2017; Devlin et al. , 2019) , they subsequently became ubiquitous in computer vision models as well  (Dosovitskiy et al. , 2021). To process an image, it is first cut into a sequence of P × P × C patches. These patches are then projected into the transformer input dimension, either using a single linear layer, or using a full-fledged CNN architecture. The output of a transformer has the same length as its input.
CLIP and VisualBERT The multi-modal features are extracted from the caption's OCR and the meme image using two vision-language models, CLIP and VisualBERT. CLIP (Contrastive Language-Image Pretraining,  Radford et al. , 2021)  is trained using text as supervision to encode images, with 400 million image-text pairs available on the internet. The training task is to predict which text is associated with an image, from all text snippets of the batch, using a contrastive objective instead of a predictive one for computational efficiency. CLIP trains an image encoder and a text encoder jointly, maximizing the cosine similarity of the image and text embeddings in the joint representation space for positive pairs, and minimizing similarity of negative pairs. The strength of this task is to offer large robustness and zero-shot capability to the model, to transfer to many classification tasks.
Image encoding is done using a variation of the Vision Transformer (ViT,  Dosovitskiy et al. , 2021). Text encoding is done using a GPT-like language model  (Radford et al. , 2019). 2 imilar to CLIP, we use a VisualBERT model  (Li et al. , 2019)  trained on visual commonsense reasoning and image captioning. VisualBERT uses self-attention to align parts of the text with regions of the image and build a joint representation. It mostly differs from CLIP in its training procedure in three phases: task-agnostic pre-training, taskspecific pre-training, and task-specific fine-tuning. Moreover, VisualBERT does not include an image encoder; the patch features are extracted beforehand with pre-trained image classification and segmentation models. We extract features using  FasterRCNN (Ren et al. , 2015) , EfficientNet  (Tan and Le, 2019)  and VGG  (Simonyan and Zisserman, 2015).
, ℓ k-1 ] T refers to the list of previous tokens. To evaluate this model, the loglikelihood of the possible labels are summed along sequence length: r = arg max r P (r | o, i) ∝ k P (ℓ (r) k | ℓ (r) <k , o, i), where ℓ (r) designates the list of tokens for the label r, such as [vil, lain] T. Additional features As explained in Section 2, our task is quite different from most multimodal tasks on which the encoders were trained; it is much more abstract and requires a lot of additional background knowledge. Thus, when using CLIP and VisualBERT, we add supplementary features as input to the classification model (MLP and attention). We add as textual features the list of entities associated with the meme, this list is directly available in the dataset. We encode the entities' names  using the same encoder as the system (CLIP or Vi-sualBERT).
Dealing with Class Imbalance The dataset faces a large class imbalance, with the class other being over-represented (78% in the train set) and classes hero and victim consisting of only 2. 7% and 5. 2% of the train set respectively. Thus, training on the raw dataset might lead to overfitting and over-predicting the majority class. Moreover, recall that the evaluation metric is Macro-F 1 , which weighs each class equally; hence the importance of solving the class imbalance issue. Our first solution was to weight labels in the loss. This loss penalisation led to poor performances; we suspect this is due to the working of the optimization algorithm we used. Adam and its variants estimate the distribution of the gradients using exponential moving averages; these estimates are faulty when the magnitude of the loss changes often.
Ensembling In order to further improve our results, we build several ensemble of our models. We filter-out models with a low validation macro-F 1 and experiment with several ensembling techniques. Due to the small size of the dataset, we did not create an additional split to evaluate our ensembling approach. In this context, overfitting the validation set is a risk. Two of the ensembling methods we evaluate are therefore non-parametric. These non-parametric strategies take the average or the median probability assigned to each class by all models. Preliminary results indicate that training a linear model to weight the output of our various models is tedious and does not improve over nonparametric strategies. We therefore turn towards gradient boosted trees  (Friedman, 2001)  trained by XGBoost  (Chen and Guestrin, 2016).
Experimental process The train set consists of 17 514 (meme, entity) pairs, the validation set 2 069 pairs and the test set 2 433 pairs. We did all the training on the datasets from the two domains, COVID-19 and US politics jointly. The test set contains examples from both domains. The evaluation is done with Macro-F 1 score; the OCR and the list of entities are provided along with the image of the meme. We run all experiments 5 times to check for the robustness of results and perform statistical testing. For CLIP, we use the biggest L/14 CLIP-ViT model built on the Vision Transformers  (Dosovitskiy et al. , 2021). Both preliminary self-supervised fine-tuning and fine-tuning while doing the classification failed. This is probably due to the size and the format of the shared task dataset, much smaller and quite different from the training data of the pretrained model; any fine-tuning leads the model to forget the knowledge it learned during pre-training.
Consequently, we freeze all layers and tune only the classifier, with the architectures described in Section 4. For VisualBERT, we fine-tune the visualbert-vcr-coco-pre model trained on caption generation and visual commonsense reasoning. For OFA enc pool with an MLP classifier, we obtained better results by fine-tuning the whole model from the vqa_large_best checkpoint 5  using a small 0. 1 label smoothing and feeding the OCR and entity both to the encoder -along with the image -and to the decoder. Our OFA seq2seq model follows the same setup using the ofa_base checkpoint. In the dataset, several entities are associated with more that one label. As this situation is infrequent, we consider the small amount of samples with multiple labels does not warrant a full-fledged multilabel classification setup.
Qualitative analysis We extract the embeddings of all entities in the train set as their are embedded by the CLIP model, right before being fed into the MLP or being used as query for the attention mechanism. Keeping only the ones occurring more than 30 times, we perform a PCA on their embeddings and represent the first two components in Figure  4. Each point represents an entity, its colour depends on the distribution of labels that are attributed to the entity, normalised by the global frequency of each label in the full dataset. We keep only the two most frequent labels associated with the entity for colouring. We can see that inanimate objects tend to be labeled as other. On the other hand, large political parties are nearly always portrayed as villain with America as a victim. The somewhat unexpected heroic status of the libertarian party can be explained by the pres- Figure  4 : PCA of entity embeddings from CLIP.
The explained variance is 33%+18%. The entities appearing more than 30 times, with labels attached to the 16 most frequent ones. The color of the embeddings reflect the role attached to the entity in the train set  ( hero, villain, victim, other). When the entity is assigned different roles, the color are mixed together; e. g. covid19 appears twice as often as other as it does as villain. ence of advertisements in the form of memes in the dataset. We can see that CLIP was able to separate the entities according to their probable class even before processing the meme. Still, the model can't clearly distinguish between most heroes and villains without seeing the meme, which is to be expected.
Discussion The multimodal aspect is crucial in this task. When looking at entity names, only 15% have an exact surface form match in the caption's OCR; moreover, the OCR is often incomplete or noisy (see example in Figure  1  with the "Exit" sign popping in the middle of the caption). Thus, using only the text is far from sufficient. On the other hand, recognising the entities in the image of the meme is not an easy task. As stated in the introduction, the image and the text are often not directly related. Moreover, the image often contains elements not seen in common image datasets; for example, meme creators often perform montages like swapping faces and objects. Overall, a lot of commonsense and cultural knowledge is needed for the model to understand what the meme is about. The absence of contextual information also makes the task difficult for humans.
To evaluate the difficulty of the task, we performed human annotation of a sample of 100 (image, entity) pairs with five annotators. Details of annotation process can be found in Appendix A. The average pairwise Cohen's κ  (Cohen, 1960) , used to measure the inter-annotator agreement, is 0. 47. It indicates a "moderate" agreement according to  Cohen (1960). However, it also shows that less than one third of the annotations are reliable  (McHugh, 2012). Moreover, the macro-F 1 scores are relatively low: the average is 0. 65 and the maximum 0. 69. Having metadata such as source website and date of publication of the meme would help human and algorithmic annotators alike. Finally, from a real-world point of view, this task is not entirely complete: the OCR and the list of entities are already provided in the dataset, and we only have to perform the classification.
Conclusion In this work, we propose several systems to solve the task of classifying entity roles in memes. We focus on comparing classification models -MLP, Attention and Seq2seq systems -on top of pre-trained multimodal encoder: CLIP, VisualBERT and OFA. Our best standalone system uses the CLIP encoder with MLP classifier, but our best score is obtained using ensembling of a large number of models. We also compare several sampling strategies to deal with the class imbalance issue, proposing dynamic sampling methods that outperform the standard uniform ("macro") sampling. As a preliminary future work, more or less straightforward processing can be performed on the dataset, at the entity-level (using an entity linker to resolve surface forms to entity identifiers, e. g. merging entities "US" and "United States" together); at the OCR-level (performing lexical normalization  (Samuel and Straka, 2021)  to deal with OCR errors and meme-specific syntax); and at the image-level (removing the text from the image, for a less noisy image embedding).
Acknowledgments We want to express our strong gratitude to  Matt Post  for the time he took providing manual annotation for our validation process. We also warmly thank the reviewers for their very valuable feedback. This work received funding from the  European Union 's  Horizon 2020 research and innovation programme  under grant agreement No. 101021607  and the last author acknowledges the support of the  French Research Agency  via the  ANR   ParSiTi  project ( ANR16-CE33-0021 ).
As recording and analysis technologies improved, the types of information that could be extracted from recordings of animal sounds increased. Presently, species-level identification is possible in most cases, and depending on the focal animals the age, sex, reproductive status, behavior, activity patterns, and even health of an individual may be estimated from acoustic recordings. Acoustic data can be used to estimate the population density of vocal animals, and dialects can indicate the geographic boundaries of a population. However, density estimation by acoustics is still in its infancy, and will require further advancement in the spatial analysis of the acoustic environment by using multiple sensors to become reliable and widely applicable. At the community level, the entire acoustic environment or soundscape can be used to estimate species abundance and biodiversity.
Recording to a Computer In the 1990s, the first sound-acquisition boards for personal computers became available, which revolutionized the way scientists collect and analyze acoustic data. Once a sound was recorded in a digital format, recordings could easily and without degradation be transferred to a computer, stored, edited, copied, distributed, played, processed, and analyzed with different algorithms. Software (either freeware or commercial) that can be used on a laptop provides scientists with "a bioacoustics laboratory in a bag. " The consumer and professional market offer a large number of sound interfaces, to be connected by USB or other standards to a PC, which can offer very high audio quality and multiple input/output channels. Smaller versions of such a setup, or compact single-board computers costing few tens of US dollars, are being used in autonomous stationary and mobile recording systems, which allow data collection and realtime data processing in remote areas for months at a time (e.
Since then, new products have been proposed by companies and research groups, with increasing performances and autonomy. These can be programmed to record at defined intervals (e. g. , every day across the dawn and dusk periods) or more regular sampling schedules (e. g. , 1 minute every 10 minutes, or 10 minutes every half-hour) to sample temporal patterns of variation in a soundscape. This way, the acoustic behavior of animals of interest can be recorded without disturbance by the recordist and for extended periods, both day and night. These recorders need to be rugged and reliable to be deployed in harsh environments. The period of time that recorders can collect data depends on the combination of available battery power and memory. Depending on these factors, terrestrial recorders can operate for weeks to months.
Once detached, the tag floats to the surface and transmits a radio signal to aid recovery. This kind of technology (Fig. 1. 21) has enabled important research on sound usage and behavioral responses of animals to anthropogenic sounds, such as naval sonars  (Tyack 2009; Tyack et al. 2011). Often a variety of sensors can be attached to the animal to provide additional environmental or behavioral data to accompany acoustic recordings. Evans et al. (2004)  attached a waterproof video camera with a hydrophone, VHS recorder, and depth-sensor to examine vocal behavior during dives of Weddell seals in Antarctica. Each time the seal vocalized, the depth and time of the sound were documented, audio and video were recorded, and the call type was later analyzed in the laboratory. Researchers had to retrieve the VHS tapes, but this species remains close to a colony during the breeding season, hauls out on the ice daily, and is easily (re)captured for recovery of the tag and data.
1. 6 Advances in Sound Analysis Hard-and Software The most important advancements in sound analysis equipment were the transition from analogto-digital systems, along with the transition from hardware to software signal processing. This provided lightweight, field portable, batteryoperated units with higher storage capacity, more stable storage media, and broadband analysis, often at a more affordable price than before. Now, even a smartphone can produce a spectrogram in real-time. Another important breakthrough was the ability of scientists to share digital data using the internet and shared storage in the cloud. Initially, the basic analysis of acoustic signals was done using oscilloscopes. These instruments provided a visual representation of the waveform of acoustic signals known as oscillograms, which are plots with amplitude on the y-axis and time on the x-axis.
1. 26). This very expensive unit was able to analyze and display stereo signals in realtime up to 32 kHz. Either reel-to-reel or cassette recordings could be used as an input, and the unit had a thermal-paper printer for printing grayshaded spectrograms. Digital sound storage and analysis became widespread given the improvements in digital computer technology and data storage, coupled with the proliferation of personal computers, and the development of dedicated sound analysis software packages. These advances also fostered the development of high-quality electro-acoustic and musical equipment (microphones, recorders, and AD-converters) for a rapidly expanding consumer market of musicians and music enthusiasts. Among the first analysis software dedicated to bioacoustics, it is worth to mention Canary, developed for Macintosh computers at Cornell University, then replaced by Raven, 16  a multiplatform software developed from the same university.
For an overview of computer-based bioacoustics sound analysis and related algorithms, see  Hopp et al. (1998 ), Zimmer (2011 ), and Sueur (2018). Many academic institutions and companies started to develop software programs for PC, Mac, and Linux computers. 17 hese software programs allowed for easy recording, manipulation, analysis, and display of signals. Now, researchers are able to collect huge acoustic datasets, and computational bioacoustics faces the Big Data problem. The latest software programs, either commercial or open source, also enable the user to run sophisticated detection/ classification algorithms over long-term data sets for automated detection of occurrences of a target sound (see Chap. 8 on detection and classification methods). This saves much time and avoids having to view and listen to the entire recording manually.
Scientists also can use readily available programming environments (including MATLAB, Octave, Python, R) to develop their own analyses, often facilitated by libraries of procedures dedicated to sound processing and bioacoustic analysis (e. g. ,  Sueur et al. 2008; Sueur 2018; Ulloa et al. 2021 ). In the late 1990s, smartphone technology was developed, along with sound analysis software for these devices. Smartphones of the twentyfirst century have the same computing power as a desktop PC. Sound recording and visualization applications were developed for both Android and iPhone Operating System (iOS) platforms. In addition, the development of the Internet of Things and low-cost computer platforms (e. g. , Arduino, Raspberry PI, and others) have allowed scientists to build web-enabled data recording and analysis systems.
The properties of the whole device are computed from predictions at the pore level and from global structural properties, such as the porosity and the specific surface area. Molecular-scale information on the confined electrolyte can be obtained from in situ X-ray and neutron experiments. 233, 306 n the theoretical side, pore-level predictions are typically obtained from a simplified description of the interface using Poisson-Boltzmann (GCS) theory, possibly also introducing excluded volume effects or using cDFT. For nanoporous materials, an alternative approach successfully used to model experimental data in the context of CDI, the so-called modified Donnan (mD) model, considers instead the equilibrium between ions inside the micropore treated as homogeneous (beyond ions adsorbed in a Stern layer) and the bulk reservoir.