dataesr/scientific-paragraphs-categorization huggingface dataset:
dataset in percent. classesfor a total of 59551 images and its testing set is composed of 11316 classes for a total of 60502 images. Finally, the In-Shop Clothes Retrieval training set is composed of 3997 classes for total of 25882 images and its testing set is composed 3985 classes and it is split into query set of 17218 images and a collection set of 12612 images. We report the Recall@K which evaluates, for a given query, if there is at least one image with the same label in the top-K retrieved images. We use K ∈{1, 2, 4, 8, 16, 32} for Cub-200- 2011 and Cars-196, K ∈ {1, 10, 100, 1000} for Stanford Online Products and K ∈ {1, 10, 20, 30, 40, 50} for In-Shop Clothes Retrieval. We report the results forCub-200-2011 and Cars- 196in Table1 and in Table.
dataset in percent. classesfor a total of 59551 images and its testing set is composed of 11316 classes for a total of 60502 images. Finally, the In-Shop Clothes Retrieval training set is composed of 3997 classes for total of 25882 images and its testing set is composed 3985 classes and it is split into query set of 17218 images and a collection set of 12612 images. We report the Recall@K which evaluates, for a given query, if there is at least one image with the same label in the top-K retrieved images. We use K ∈{1, 2, 4, 8, 16, 32} for Cub-200- 2011 and Cars-196, K ∈ {1, 10, 100, 1000} for Stanford Online Products and K ∈ {1, 10, 20, 30, 40, 50} for In-Shop Clothes Retrieval. We report the results forCub-200-2011 and Cars- 196in Table1 and in Table Feat. + Post-att. N 2 4 8 16 R@1 53. 4 ± 0.
As a consequence of this ambiguity and of the impractical aspects of 2 D drawing for computerization, this type of representation cannot be directly used as a computer formalization for animating an avatar. Videos recordings are another, more precise and exact representation of signs that is very popular in the linguistic community Stokoe  [32]  HamNoSys  [26]  SLPA  [37, 39]  SignWriting  [33]  Scripting Language SiGML (extended)  [42]  QualGest  [25]  Losson  [43]  Zebedee  [44]  EMBRScript  [45]  Fidelity : absence of ambiguity, fidelity to the original movement, precise description of the sign, preservation of the intent of the signer. Temporal aspects, synchronization : the dynamics of the movement is specified, the synchronization between the different channels is managed. Non manual features : the status of non-manual components (facial expressions, gaze, etc.
[GLOSS]). A sequence of glosses is not a translation but the oral language description of a sign language utterance: for example, a sentence glossed as "[YOU][LIVE][WHERE?]" will be translated as "Where do you live?". Furthermore, a sign may not have a one-word equivalent in an oral language. In this case, the corresponding gloss can contain more than one word (e. g. , [TO GIVE] may be used in different contexts, such as in the gloss description: "[TO-GIVE-A-PAPER]" or "[TO-GIVE-A-GLASS]". The way to execute a sign is not indicated in the gloss description. Therefore, few animation systems can choose to rely exclusively on a glossing description of the desired SL production since it implies the presence of a motion database annotated on the same gloss-level as the specification like in  [28]  for motion capture data or in  [29]  for hand-crafted animation.
It was designed to be language-independent and extensible to new linguistic mechanisms. It can better handle some SL mechanisms like symmetry (the symbol. at the beginning of the transcription in Fig. 7  ) and can be transcribed in a linear way using computer Unicode symbols. However, it is still limited in terms of non-manual features. SignWriting  [33]  is based on a dancing gestures transcription and constitutes a more graphical and intuitive notation of SL. It integrates some facial expressions, body movements and even iconic signs (see Fig. 7  ). Hand configurations and orientations can be defined using a limited set of symbols but the placement of the hands with respect to the body is indicated only in a relative manner thus leaving some ambiguities. It differs from the two previous notations in the sense that it was designed to be used by deaf people as a writing system and not by linguists or researchers as a notation system.
It is SL independent and is taught in deaf classes around the world  [34]. Other writing systems exist for different sign languages (like "SEL" for Brazilian Sign Language  [35]  ) but none as popular as SignWriting. Stokoe's notation , HamNoSys and SignWriting focus on dealing with the representation of the static components of SL but fail to represent the dynamics of signs and synchronization of the different components. The temporal aspects of SL were thoroughly studied in the work of Johnson and Liddell. In 1989, they introduced the Movement/Hold model  [36]  where signs were defined as an alternation of static poses ( Hold ) and dynamic transitions ( Movement ) between two consecutive poses. Then, between 2010 and 2012, they defined the Sign Language Phonetic Annotation (SLPA)  [37]  which relies on the Posture-Detention-Transition-Shift (PDTS) classification.
Descriptive languages based on existing notation systems Descriptive markup languages are used to describe and structure a document or a data set. Among them, the eXtensible Markup Language (XML) describes the data in a way that is understandable both by the humans and the computers. It is commonly used to describe natural languages and is suited to the specification of signs as it can integrate the information of existing notation systems in a computer-readable language. The SignWriting Markup Language (SWML)  [46]  is an XML version of SignWriting and has been designed to allow storage and processing of sign language files. The structure corresponding to a sign, called a signbox , contains the exact same information as the SignWriting transcription of the sign. Consequently, it does not overcome the limitations of SignWriting , namely the ambiguities of the hand placement and the time management.
Programming languages for sign synthesis The definition of a programming language for synthesizing SL implies the specification of a dedicated lexicon and syntax to be used in subsequent instructions. Such languages are often defined for a specific animation engine and are less generic than descriptive languages but offer more freedom and flexibility to the programmer. In an early work, Lebourque et al. [51]  defined QualGest , a high-level specification language dedicated to LSF that takes into account the four manual parameters (hand configuration, placement, motion and orientation), called gestems. To specify the hand placement, they use a discretization of the signed space, including the definition of: (i) a set of directions (defined from the three main planes -sagittal, frontal and horizontal-, plus two intermediate planes), (ii) of amplitudes (proximal, medial, distal and extended), and, (iii) of body positions.
Repetitions in the movement, synchronicity properties of the hands, symmetry or anti-symmetry characteristics, or relative placement of the hands can be specified. In addition, Losson's representation uses a parametric computer language that allow the specification of sign inflection mechanisms (size and shape specifiers, spatial referencing). Fig. 10  shows an example of Losson's specification for the sign [CHIMNEY] in LSF using two shifts. In QualGest and Losson's approaches, the time does not explicitly appear in the SL specifications, whereas the EMBRScript is based on the explicit specification of key poses in absolute time, allowing a fine temporal control of the animation  [52]. It was origi- nally designed to describe the motions of Embodied Conversational Agents (ECA) and was extended to be applied on SL avatars by adding new hand configurations, facial expressions and gaze directions  [45].
Hand-Crafted Animation In hand-crafted animations, the specification of the keyframes is done manually. 3D traditional animation consists in setting the avatar in a specific pose at different frames of a timeline using a specialized animation software such as Autodesk Maya  [53]  or Blender  [54]  and the knowledge of human anatomy and motion. Different joint angles values are tested and the best values are selected for a particular pose of a particular sign. The process is fastidious and cannot be generalized to other signs. The early work of Shantz  [55]  in 1982 which is considered to be the first work on sign language avatar animation  [56]  , used this technique. 3D rotoscoping is a particular instance of 3 D traditional animation. It is used to produce realistic movements from video footage.
Data-driven sign synthesis Data-driven synthesis techniques use captured motions to animate an avatar. They are less exploited than hand-crafted or procedural synthesis techniques even though they provide a high level of realism that can hardly be achieved by procedural means. Motion Capture (MoCap) is used to record the position of markers placed on a human being performing a motion. The position and orientation of the human's joints is then deduced from the MoCap data during the post-processing step. Fig. 16  describes the offline motion database creation process. In this section, we are only interested in the synthesis methods based on a MoCap database. However, the annotation step which consists in dividing the continuous flow of motion into smaller segments ( segmentation ) and labeling those segments, directly impacts the motion synthesis.
It is technically impossible to capture all the signs in all the possible contexts due the size of the SL vocabulary, the multiple inflection mechanisms of signs and sentences in SL, the need for various Deaf participants, and time and memory constraints. Therefore, synthesizing signs using a limited set of pre-recorded signing sequences is the major challenge of data-driven techniques. Three main approaches exist to create isolated signs using motion capture data: 1. Play-back : the captured data can be played-back without modification, 2. Editing : new motion can be created by editing existing motion data, 3. Machine Learning : new data can be synthesized using knowledge from MoCap data via machine learning approaches. Regardless of the chosen synthesis technique, data-driven synthesis involves motion retrieval.
It consists in choosing and extracting the best motion(s) for a particular application among a set of motions. It can be done by (i) directly querying the motion features (e. g. , see Kapadia et. al. [85]  for motion retrieval of non linguistic motion using Laban movement analysis), or (ii) using a textual search in a semantic database containing the annotation of the motion  [83, 86]. In this second case, the retrieval process relies on a management system database, separating the two levels of representation (semantic symbols and raw motion data) (see Fig. 17  ). This is the most straightforward technique as the motions of sign language are semantically meaningful. However, motion feature query remains interesting to decide between several motions with the same annotation label  [87]  , or to extract motions at a finer level than gloss.
Assuming that the quality of the captured database and postprocessing is correct, the play-back of a pre-recorded sign will give an accurate and realistic motion. Indeed, the main advantage of MoCap technologies is that it preserves the human qualities of motion. The motion and dynamics of the generated sign will therefore be perceived as natural and credible, thus increasing the acceptance of an avatar driven by such a method  [88]. However, while this technique can be used to generate new utterances by concatenating existing signs as we will see in Section 5. 2. 1 , the generation of new signs, not existing in the original database, or the editing of a sign in order to adapt it to a new context are not the objectives of the play-back approach. The utterances produced using this technique are therefore limited by the number and variety of the recorded data.
The existing database can also be augmented by editing the pre-recorded motion using synthetic animation techniques. For example, in  [83]  , some signs are created by inverting the direction of the hand motion or by modifying the hand configuration of an existing sign. The sign  [DARK]  in French Sign Language can be created by inverting the hand trajectory of the sign [LIGHT] (see Fig. 19  ) whereas the object of the sign [TO GIVE] can be modified by changing the hand configuration (see Fig. 5  ). Motion warping is also a motion editing technique that can be applied to add variability in sign language generation. It consists in altering a motion of the database by changing its trajectory while keeping the kinematic properties of the motion  [91]. This is a promising technique for data-driven synthesis of SL, but Fig.
18. The Sign360 application of MoCapLab  [89, 90]. so far little work has been dedicated to the development of this technique. Another approach, Dynamic Time Warping (DTW), can be used to introduce temporal variability in the sign language production. This technique can be used: i) as an elastic distance to retrieve motions in the database; ii) as a synthesis process to stress/compress the edited motion; iii) as a process to generate style transfer whether at a phonological level  [92]  , or at an utterance level  [93]. In terms of style transfer, Gerard, the avatar of  [94]  , could generate stylized sign sequences in LSF by learning the time warping of other recorded sequences. As data-driven techniques, motion warping and DTW ensure natural-looking kinematics of the resulting movements.
However, the realism of the transformed and edited movements has still to be verified with respect to the generated postures and the meaning of the synthesized sentences. Machine learning methods are another way to reuse and generalize motion data. It relies on the captured data to extract knowledge and models that can provide new plausible and contextualized movements. Carreno et al. [95]  present a very thorough survey on motion synthesis based on machine learning methods. For the particular case of sign language synthesis, Huenerfauth et al. [16]  studied and synthesized directional verbs like [ASK], [MEET] or [SEND] using MoCap data of SL performances from fluent signers as a training data set to learn their synthesis models. Examples of data-driven avatars: • Tessa  [96]  , a British Sign Language avatar, and Simon  [97]  , a Sign Supported English 6  avatar both take advantage of the play-back technique.
• Play-back of signs in Swedish Sign Language was also done by Alexanderson et al. [98]  , but the main concern of their work was the study of motion capture and skeleton reconstruction. • The Sign3D project combines both play-back and sign synthesis editing techniques  [28]. • For commercial applications, Sign360 of the French company MoCapLab presents a French Sign Language avatar driven by pre-recorded gestures  [89, 90]  (see Fig. 18  ). Data-driven synthesis techniques are effective ways to produce natural looking motions. However, the quality of the resulting avatar animations depends on the granularity of the annotation and on the size and content of the initial corpus. Editing techniques to generate new signs are still rare in the SL animation field.
Limitation of the representation with a sequence of glosses Many machine translation systems describe an utterance as a simple sequence of glosses. As the order and nature of the signs is given by this representation, it is suited to concatenative synthesis provided the presence of a sign database annotated on the same gloss-level as the specification. However, utterance representation is a complex problem for which a sequence of glosses is not the appropriate solution as cooccurrences of signs are common phenomena. The main challenges of utterance representation are listed here. We illustrate each challenge with an example that highlights the limitations of the gloss representation.
It can be represented using different gloss sequences which will have an impact on the resulting animation. With a sequence of simple glosses, this would give: [I][HE][SMALL][BALLOON][GIVE] (5) This would certainly result in the concatenation of many isolated signs in their citation form. The utterance would not be grammatically correct. However, the use of parameterized glosses can cover this type of inflection: [ BALLOON _ small ][ GIVE _ O con f ig _ I → he ] (6) The representation (6) only contains two glosses, a first gloss naming the specified object and a second gloss corresponding to an inflected sign that can be retrieved from a database or generated on-the-fly. This parameterized representation will certainly achieve a result more similar to what a real signer would do. However, it implies a very large database containing an important number of inflected signs or a synthesis engine capable of understanding the syntax of the representation and of creating the corresponding sign (e.
Timing Information There are two philosophies in timing management for signing avatars. This timing can be (i) indicated explicitly, in a relative or absolute manner, in the representation of the utterance at the input of the synthesis model (in this case, it is an additional constraint for the model)  [45]  , or (ii) computed by the synthesis model according to the constraints of the system  [28, 60]. For example, if the sentence is a concatenation of movement segments present in a database, the timing will be constrained by the length of these segments. Sequence of gloss representations, whether parametric or not, do not account for the timing but only for the linear order of the signs.
Utterance synthesis approaches Two main utterance synthesis approaches are often distinguished: the concatenative and the articulatory synthesis  [64]  (see Fig. 27  ). In the case of concatenative synthesis, the utterance objective is considered as a set of smaller objectives. Concatenative synthesis involves a motion/sign database and consists in concatenating small chunks of existing data while articulatory synthesis computes the sign language utterance directly from the gesture specification. Hybrid avatars with animation systems taking advantage of both techniques are also emerging.
Concatenative synthesis It is the process in which chunks of pre-recorded or presynthesized motions are concatenated successively and/or on the different SL channels (see Fig. 28  ). Motion interpolation or blending techniques are used to build the transitions between two chunks of motions. Motion blending consists in doing, at each frame, an interpolation of motions present in the database to create new motion having the characteristics of the initial motions. The quality of the result strongly depends on the length and granularity of the chunks of motion. The simplest and most common approach is to consider motions at a gloss-level, but smaller motion chunks can also be used, leading to a more precise control of the avatar animation. Motion synthesis is therefore based on an annotated database of signs (or finer grained motions) that is queried to compose the utterance.
The annotation of the data is essential: the granularity of the synthesis is restricted by the granularity of the annotation, and the presence or absence of artifacts in the final animation depends partially on the quality of the segmentation of the data. The database is built and annotated offline while the sign concatenation can be done online. Utterance representations based on sequences of glosses are particularly suited to this synthesis technique. The following paragraph details the differ- ent avatars technologies that make use of concatenative synthesis for utterance synthesis depending on the nature of the content of the original database.
Using Hand-Crafted Animations In this case, concatenative synthesis relies on a database of hand-crafted motions and on motion interpolation to smoothen the transitions between the concatenated gestures. Paula  [57]  and Elsi  [29]  , the avatars presented in Section 4. 3. 1 , rely on this principle. Paula 's motion is created in real-time by automatically combining and possibly modifying the hand-crafted key postures in accordance to the desired sign language utterance. Elsi can sign full utterances created by concatenation of isolated signs designed by rotoscopy. Unfortunately, she can create very few novel utterances as rotoscopy does not provide for context-dependent signs. However, this method suits the application of Elsi who does not need to convey novel messages since she has to provide information in train stations to deaf passengers by combining pre-recorded fill-inthe-blank sign language utterances and isolated signs.
Using Automatically Generated Signs It is also possible to build a sign language utterance by editing and concatenating signs extracted from a database of automatically keyframed or of procedurally synthesized signs. Annotation tags are queried to retrieve the relevant keyframes/signs from the database. The avatars relying on automatic keyframing for the generation of isolated signs ( Section 4. 3. 1 ) often synthesize utterances by simple concatenation of whole signs. Examples of avatars using concatenative synthesis with automatically generated signs: • The EMBR avatar of Kipp et al. [45, 52]. • The avatar of Krnoul et al. [66]. • The parametric avatar of Irving et al. [62]. • The eSIGN avatar  [111].
Using MoCap Data Concatenative synthesis on MoCap data consists in combining and concatenating previously captured motion chunks (often corresponding to a sign or gloss segment). The transition between the motion chunks can be done using different kinds of interpolation and blending methods  [24]. This concatenative synthesis produces realistic motions on play-back sequences but the quality of the utterance as a whole strongly depends on the quality of the synthesized transition segments. Moreover, the signs will be played by the avatar the exact same way as they were recorded. It means that the objects described in the new utterance will have the same aspect as the ones recorded. The creation of novel utterance is thus limited by the motions available in the database. Examples of avatars using concatenative synthesis with MoCap data (see Section 6 for more details) : • The Sign3D avatar  [28].
Articulatory synthesis One of the drawbacks of concatenative synthesis, regardless of the nature of the database, is that the input is often a sequence (of glosses or motions) while co-occurring spatial phenomena (e. g. , proforms or iconicity) are often present in sign languages and cannot be fully captured by concatenation alone. Therefore, another way to build utterances in sign language is to create them onthe-fly following an utterance specification and not depending on a predefined database of fixed motion chunks. Iconicity, proforms, transitions between the signs or co-occurring phenomena can be incorporated directly into the sign description so as to take the context into account. Sentences build this way can be precisely controlled but the work of describing the utterance using a sign or a gesture specification (taken as input of such a system) can be extremely tedious.
Hybrid synthesis Hybrid models take advantage of the strengths of concatenative and articulatory synthesis. The CUNY American research group worked on both procedural animation techniques and data-driven synthesis approaches using MoCap data and compared the two approaches in  [82]. The sign language research team of the University of East Anglia also studied both technologies, first with Tessa (concatenative)  [96]  , followed by the eSIGN project (articulatory)  [113]. They proposed to use MoCap data to add realism to their procedurally animated avatar. DePaul University research group implemented this idea on Paula , their avatar animated with hand-crafted keyframes  [58]. Moreover, Paula was recently improved with IK solvers that procedurally modify the hand-crafted animations in order to synthe-engine for SL synthesis developed for the ViSiCAST  [74]  and, later, the eSIGN  [111]  projects.