dataesr/scientific-paragraphs-categorization huggingface dataset:
E. QUALITATIVE RESULTS Some examples of results obtained on the MSRVTT corpus are illustrated in Figure  9. The results indicate that the quality of the predicted captions is affected by various factors, including the availability of audio and visual information, the complexity of the content and the accuracy of the ASR. When the transcripts are pertinent (with salient words represented in purple in Figure  9 ), combining textual and visual modalities leads to precise captioning. In contrast, in the absence of the audio or more generally when the transcript channel is not consistent with the content (transcripts represented in red in Figure  9 ), the predictions rely only on visual cues and are less informative. In some cases, the model lacks access to external knowledge, as illustrated in example 3, where specific information like the car brand remains elusive from both visual and textual cues.
To address this limitation, future research axes could explore the integration of external knowledge sources, such as databases or ontologies, offering a promising prospect for enhancing the model's contextual understanding. Another limitation arises from our offline transcript extraction, illustrated in case 9, where the model lacks awareness that the transcript represents music lyrics. This highlights a challenge where textual data can inadvertently introduce noise into the model's predictions. To mitigate this, integrating audio features emerges as a valuable strategy, not only for recognizing music lyrics but also for scenarios VOLUME XX, 2017 necessitating the recognition of environmental sounds or the capture of emotional context in videos. In general, the accuracy of the predicted captions is largely influenced by the type and quality of the input data.
In all cases, incorporating multimodal approaches can enhance the precision of the predictions. VOLUME XX, 2017 We have also shown that the gain in performance strongly depends on the nature of the data in different categories. This indicates a need for further research to develop more effective training methods that can take into account in a fine-grained manner the data characteristics of various categories. Future work could explore pre-training on larger datasets to further improve the performance of our approach. Large-scale pre-training allows the model to learn and capture the intricate correlations and interactions between different modalities. It facilitates the comprehension of complex multimodal patterns that may not be discernible in smaller, more constrained datasets. Another potential avenue can be the exploration of knowledge-augmented models.
Advantages of Transformers Since their introduction, the transformers have had a significant impact on deep learning approaches. In natural language processing (NLP), before transformers, most architectures used to rely on recurrent modules, such as RNNs  [2]  and in particular LSTMs  [3]. However, recurrent models process the input sequentially, meaning that, to compute the current state, they require the output of the previous state. This makes them tremendously inefficient, as they are impossible to parallelize. On the contrary, in transformers, each input is processed independent of the others, and the multi-head attention can perform multiple attention computations at once. This makes transformers highly efficient, as they are highly parallelizable. This results in not only exceptional scalability, both in the complexity of the model and the size of datasets, but also relatively fast training.
Notably, the recent switch transformers  [7]  was pretrained on 34 billion tokens from the C4 dataset  [8] , scaling the model to over 1 trillion parameters. This scalability  [7]  is the principal reason for the power of the transformer. While it was originally introduced for translation, it refrains from introducing many inductive biases, i. e. , the set of assumptions that the user makes about the structure of the model input. In doing so, the transformer relies on data to learn how they are structured. Compared to its counterparts with more biases, the transformer requires much more data to produce comparable results  [5]. However, if a sufficient amount of data is available, the lack of inductive bias becomes a strength. By learning the structure of the data from the data, the transformer is able to learn better without human assumptions hindering  [9].
In most tasks involving transformers, the model is first pretrained on a large dataset and then fine-tuned for the task at hand on a smaller dataset. The pretraining phase is essential for transformers to learn the global structure of the specific input modality. For fine-tuning, typically fewer data suffice as the model is already rich. For instance, in natural language processing, BERT  [10] , a state-of-the-art language model, is pretrained on a Wikipedia-based dataset  [11] , with over 6 million articles and Book Corpus  [12]  with over 10,000 books. Then, this model can be fine-tuned on much more specific tasks. In computer vision, the vision transformer (ViT) is pretrained on the JFT-300M dataset, containing over 1 billion labels for 300 million images  [5]. Hence, with a sufficient amount of data, transformers achieve results that were never possible before in various areas of machine learning.
Second, transformers suffer from lack of inductive biases. Since CNNs were specifically created for vision tasks, their architecture includes spatial inductive biases, like translation equivariance and locality. Therefore, the transformers have to be pretrained on a significantly large dataset to achieve similar performances. The vision transformer (ViT)  [5]  is the first systematic approach that uses directly transformers for vision tasks by addressing both aforementioned issues. It rids the concept of convolutions altogether, using purely a transformer-based architecture. In doing so, it achieves the state of the art on image recognition on various datasets, including ImageNet  [17]  and CIFAR-100  [18]. Figure  4  illustrates the ViT architecture. The input image is first split into 16 × 16 patches, flattened, and mapped to the expected dimension through a learnable linear projection.
The resulting tokens are fed into a transformer, and finally the resulting classification token is passed through an MLP to compute output probabilities. Figure inspired from  [5]  Due to the lack of inductive biases, when ViT is trained only on mid-sized datasets such as ImageNet, it scores some percentage points lower than the state of the art. Therefore, the proposed model is first pretrained on the JFT-300M dataset  [19]  and then fine-tuned on smaller datasets, thereby increasing its accuracy by 13%. For a complete overview of visual transformers and follow-up works, we invite the readers to study  [9, 20].
Improvements over the Vision Transformer In this section, we present transformer-based methods that improve over the original vision transformer (Subheading 2. 3) in two main ways. First, we introduce approaches that are trained on smaller datasets, unlike ViT  [5]  that requires pretraining on 300 million labeled images (Subheading 3. 1). Second, we present extensions over ViT that are more computational-efficient than ViT, given that training a ViT is directly correlated to the image resolution and the number of patches (Subheading 3. 2).
Data Efficiency As discussed in Subheading 2. 3, the vision transformer (ViT)  [5]  is pretrained on a massive proprietary dataset (JFT-300M) which contains 300 million labeled images. This need arises with transformers because we remove the inductive biases from the architecture compared to convolutional-based networks. Indeed, convolutions contain some translation equivariance. ViT does not benefit from this property and thus has to learn such biases, requiring more data. JFT-300M is an enormous dataset, and to make ViT work in practice, better data-efficiency is needed. Indeed, collecting that amount of data is costly and can be infeasible for most tasks. Data-Efficient Image Transformers (DeiT)  [21]  The first work to achieve an improved data efficiency is DeiT  [21]. The main idea of DeiT is to distil the inductive biases from a CNN into a transformer (Fig.
The teacher network is a Convolutional Neural Network (CNN). The main idea is that the distillation head will provide the inductive biases needed to improve the data efficiency of the architecture. By doing this, DeiT achieves remarkable performance on the ImageNet dataset, by training "only" on ImageNet-1K  [17] , which contains 1. 3 million images. Convit  [22]  The main disadvantage of DeiT  [21]  is that it requires a pretrained CNN, which is not ideal, and it would be more convenient to not have this requirement. The CNN has a hard inductive bias constraint that can be a major limitation. Indeed, if enough data is available, learning the biases from the data can result in better representations. Convit  [22]  overpasses this issue by including the inductive bias of CNNs into a transformer in a soft way.
Specifically, if the inductive bias is limiting the training, the transformer can discard it. The main idea is to include the inductive bias into the ViT initialization. Therefore, before beginning training, the ViT is equivalent to a CNN. Then, the network can progressively learn the needed biases and diverge from the CNN initialization. Compact Convolutional Transformer  [23] , DeiT  [21] , and Convit  [22]  successfully achieve data efficiency at the ImageNet scale. However, ImageNet is a big dataset with 1. 3 million images, whereas most datasets are significantly smaller. To reach higher data efficiency, the compact convolutional transformer  [23]  uses a CNN operation to extract the patches and then uses these patches in a transformer network (Fig. 6 ). The compact convolutional transformer comes with some modifications that lead to major improvements.
First, by having a more complex encoding of patches, the system relies on the convolutional inductive biases at the lower scales and then uses a transformer network to remove the locality constraint of the CNN. Second, the authors show that discarding the "class" token results in higher efficiency. Specifically, instead of the class token, the compact convolutional transformer pools together all the patches token and classifies on top of this pooled token. These two modifications enable using 3. 2 Computational Efficiency smaller transformers while improving both the data efficiency and the computational efficiency. Therefore, these improvements allow the compact convolutional transformer to be successfully trained on smaller datasets, such as CIFAR or MNIST. The vision transformer architecture (Subheading 2.
The language stream is initialized with BERT trained on Book Corpus  [12]  and Wikipedia  [11] , while the visual stream is initialized with Faster R-CNN  [33]. On top of the pretraining of each stream, the whole architecture is pretrained on the Conceptual Captions dataset  [34]  on two pretext tasks. ViLBERT has been proven powerful for a variety of multimodal tasks. In the original paper, ViLBERT was fined-tuned to a variety of tasks, including visual question answering, visual commonsense reasoning, referring expressions, and caption-based image retrieval.
CLIP Connecting Text and Images (CLIP)  [35]  is designed to address two major issues of deep learning models: costly datasets and inflexibility. While most deep learning models are trained on labeled datasets, CLIP is trained on 400 million text-image pairs that are scraped from the Internet. This reduces the labor of having to manually label millions of images that are required to train powerful deep learning models. When models are trained on one specific dataset, they also tend to be difficult to extend to other applications. For instance, the accuracy of a model trained on ImageNet is generally limited to its own dataset and cannot be applied to realworld problems. To optimize training, CLIP models learn to perform a wide variety of tasks during pretraining, and this task allows for zero-shot transfer to many existing datasets.
Finally, the terminal classification token embedding is passed through an MLP to output a probability for all video classes. During inference, the final prediction is obtained by averaging the output probabilities from three different spatial crops of the input video clip (top left, center, and bottom right). TimeSformer achieves similar state-of-the-art performances as the 3D CNNs  [53, 54]  on various video classification datasets, such as Kinetics-400 and Kinetics-600  [55]. Note the TimeSformer is much faster to train (416 training hours against 3840 hours  [50]  for a SlowFast architecture  [54] ) and, also, more efficient (0. 59 TFLOPs against 1. 97 TFLOPs  [50]  for a SlowFast architecture  [53] ).
temporal phenomenon. […] [T]his efflorescence of image particles and microtemporalities within the frame risks shattering the unity and discretion of the image. Coherence has to be constructed after the event by applying massively reductive processes to the frame. Coherence is achieved then by nominating as redundant the mass of detail, color, and nuance that human observation is capable of. Instead, the task of observing is first modeled on a good-enough solution for an imaginary statistical norm of perception and then the process of selecting what to observe is automated. […]  [B]ecause each frame and each pixel or group thereof must be generated on the fly from stored datasets interpreted by software protocols, there is an active coming into existence at the level of the pixel. 65 In this sense we are dealing less than ever with stable, dependably addressable, losslessly reproducible image objects, but rather always only with distributed image processes.
Any market participant who distributes their streaming portfolio over market-dominating providers such as Akami or Amazon Web Services (AWS) brings the media server spatially closer to the edge of the network, to the client-thus reducing problems with latency times and volatile transmission rates. The economic and strategic significance of CDNs is evident, for instance, in the fact that the streaming service Netflix no longer maintains any data centers of its own (and had no problem transferring its entire database-»everything that happens before you hit ›play‹«79-into the cloud of the AWS ecosystem80 despite competing with Amazon Video), while at the same time insisting on distributing the 125 million hours of video playback per day currently being requested by customers exclusively via Open Connect, their in-house CDN.
Despite the rhetoric of wirelessness, we exist in a world that is more wired than ever«  (Starosielski, Undersea Network, 9). 100 David M. Berry, »Messianic Media: Notes on the Real-time Stream,« Stunlaw (blog), Sept. 12, 2011, http://stunlaw. blogspot. com/2011/09/messianic-media-notes-on-real-time. html. 101 »As opposed to previous mechanisms that work on opening and closing server connections, and pulling in information on request, this new type of data processing performs a continuous query for new data units that arrive in the database and pushes the result into the stream according to the filter being used. The result is thus a persistent, real-time connection between a server and a user« (Nadav Hochman, »The Social Media Image,« Big Data & Society, July-Dec. 2014, 1-15, here: 2). whose beginning has repeatedly been linked to the advent of RSS feeds102 but then expanded further through push technologies, is now considered a fundamental principle of the general streaming quality of data traffic, especially due to the dominant architecture and addressing modes used by popular platforms with social media characteristics.
103 The underlying model of real-time processing replaces one-time data bank requests with continuous queries,104 and is articulated on the surface primarily through responsive interfaces with the option of real-time interaction (or with a prompt for it: »What's happening?« asks Twitter, »What's on your mind?« asks Facebook). The continuous, dynamic implementation of data streams flowing in and out reaches the user in the form of a mediated ›nowness,‹ which may be programmed differently in terms of the media's time-critical operational logic (how is the live feed synchronized, how is the newest content integrated into the timeline in each case, how are the interaction modes operated, etc. 105), but in the final analysis mainly either generates synchronization effects, or transmits them in a way that »habitualizes,« as Wendy Chun suggests.
Datafication therefore as a rule also means: a situating incorporation into a data bank architecture. If in this regard we follow Marcus Burckhardt's theory of digital data banks, which emphasizes the role of data bank management systems (DBMS) as generative collection technologies and fundamentally rejects the idea of passive memory, at this level as well we can see a technological infrastructure of digital imagery that does not so much collect static objects as it manages dynamic processes. Instead of reproducing previously established hierarchies, »the data bank represents the potential to make it possible to arrange or experience the same information in a variety of ways on the user interface, and in doing so to give it different meanings. «133 Anything that is stored in a certain way, in a specific form, with a unique address, can be visualized, transported, communicatively mobilized in various ways.
In this sense, we do not simply access stored information-rather, it is actualized, that is, generated anew. Precisely how the data are secured in the »deep structure« of a data bank is thus independent of the »level of expression,«134 insofar as data bank architectures (as well as compression algorithms) are typical cases of »blackboxing. «135 Their power of definition is manifest in inclusions and exclusions of particular data sets, in communicating compatibility. The usefulness of data is unrelated to any understanding of its technological condition and organization on the part of the user. Such nontransparency in no way detracts from the efficacy of a data bank. The user must pragmatically recognize the compatibility settings, but need not understand the logic of storage, the data room, in order to be able to initiate productive queries, whose capacity for connection causes manifestations as well as meanings to be redistributed.
In the context of Lisa Gitelman's definition of the term ›data‹-according to which data are discretely and aggregatively, abstractly and graphically mobilizable-this situation is presented as follows: Today the ubiquitous structures of data aggregation are computational forms called relational databases. made concretely productive. Against the »etymological legacy«138-derived from datum: that which is given-data are fundamentally created on various levels, or lifted by queries and connecting operations from the entirety of a data pool, as amorphous as it is opaque: »Data need to be understood as framed and framing, understood, that is, according to the uses to which they can be put. «139 This aspect, which Gitelman interestingly describes using the terminology of photographic framing, is strongly emphasized by Sebastian  Gießmann and Marcus Burckhardt: »[…]  the fact that data always seem to be given for something: for their later interpretation, for their technological evaluation by computers, for future access, and for the understanding of reality.
«140 The relative open-endedness of data manufacturing, which is crucial to relational data banks, is thus not least restricted by the layout, the efficacy of search machine functionalities, which, particularly under the conditions of big data, not only produce all sorts of connections, but also oversight and hierarchy. Thus already from the viewpoint of standardized data collection in the form of relational data banks, the »major trend toward datafication«141 that can be observed on various levels, including in the field of the image-a spectrum that in the context of this study ranges from digital technologies of acquisition and transcoding to streaming video codecs and the data-image-sensory environmental calculations of the Internet of things-describes a dynamic transport process, guided by perspectives of evaluation.
142 Data sets here are assigned a discrete key in order to render them capable of being algorithmically processed, manifest, and distributed. Under the real-time conditions outlined here, which process and distribute ever increasing amounts of data in stream form, this fundamental mobility is naturally further intensified, thus requiring data management architectures that are aimed at transmitting and processing in real time. Christoph Engemann describes the pressure this puts on logistical infrastructures in terms of social media application contexts: ›Firehose‹ is the informal term for the dense, practically violent stream of data that aggregates in the data banks from end devices. […] The data banks are like reservoirs into which the data flows, both coming to rest there as well as escaping once again at high speed, filtered and at defined points.
143 In the ›real-time web‹ it has already been some time since the term ›streaming data‹ referred solely to the transmission of data that-as in the case of audiovisual signals-have a distinctive, concretely perceivable temporal behavior. In contrast to static data, which are managed in relational data banks, streaming data are temporally organized, continuous, unrestricted, non-persistent, as well as being comparatively loosely structured, which necessitates corresponding systems of live processing. Data stream algorithms with the capacity for sequential reading and processing therefore no longer operate in DBMS infrastructures, but rather in so-called Data Stream Management Systems (DSMS). Between incoming and outgoing streams there is a time-critical specification profile oriented toward sequential access: to manage continuously new data that is constantly being restructured, analyze it and make it accessible in real time.
pdf, 3f. ). 150 Jeffrey  Pomerantz, Metadata, Cambridge, MA: MIT Press, 2015, 17f. Another layer of metadata, which needs to be distinguished from this, does indeed adhere to the nominal image data set, but is stored as fundamentally separate, in proprietary data banks. 151 It does not derive from the context of acquisition, but rather accumulates distribution-related data-and data that result from image-sharing practices on social media, that is, for instance, individually or collaboratively generated tags152 or information-economic values such as numbers of queries and dissemination  (ratings, likes, views, retweets, etc. 153). Especially for platforms such as Facebook, images are primarily of interest as digital metadata objects, by means of which profiles and the surrounding social media network can be regulated »governmentally«154 and data mining processes can be supplied.
The digitalborn image is never singular, it appears in series, repetitions, sequences, rapid volleys  […]. Each retweet, reblog, rating or tag generates further metadata which can amplify the intensity of the image, its reproducibility, and create topologies between images. With respect to Flickr, the simple act of tagging an image ›cat‹ immediately connects the image 161 »Aiming to transform the way in which people search and browse theirs and others' photos, these systems automatically analyze and recognize searchable visual content such as objects, text, or faces and automatically add searchable tags to images and videos where those items are ›seen‹ by the software. These types of visual data management are becoming ever-more synchronous with the act of picture taking within and outside of the stream and operate both on the level of the individual's image collection and over collective datasets from many people.