Venue: NAACL 2025

# FLEURS-ASL: Including American Sign Language in Massively

Multilingual Multitask Evaluation

 Garrett Tanzer

Google

gtanzer@google.com

###### Abstract

Sign language translation has historically been peripheral to mainstream machine translation research. In order to help converge the fields, we introduce FLEURS-ASL, an extension of the multiway parallel benchmarks FLORES (for text) and FLEURS (for speech) to support their first sign language (as video), American Sign Language, translated by 5 Certified Deaf Interpreters. FLEURS-ASL can be used to evaluate a variety of tasks--primarily sentence- and discourse-level translation--between ASL and 200 other languages as text, or 102 languages as speech. We provide baselines for tasks from ASL to English text using a unified modeling approach that incorporates timestamp tokens and previous text tokens in a 34-second context window, trained on random video clips from YouTube-ASL. This model meets or exceeds the performance of phrase-level baselines while supporting a multitude of new tasks. We also use FLEURS-ASL to show that multimodal frontier models have virtually no understanding of ASL, underscoring the importance of including sign languages in standard evaluation suites.1

Footnote 1: We publicly release FLEURS-ASL at this link under CC BY-SA 4.0.

_Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 6167–6191

April 29 - May 4, 2025 ©2025 Association for Computational Linguistics

## 1 Introduction

Despite calls to include sign languages in mainstream natural language processing Yin et al. (2021), sign language translation remains peripheral to the machine translation field. The sign language translation field uses individual benchmarks for each language--such as How2Sign Duarte et al. (2021) for ASL--constructed as train/test splits from one underlying dataset, often with overlapping signers. Meanwhile, mainstream machine translation has developed independently constructed massively multilingual benchmarks like FLORES Guzman et al. (2019); Goyal et al. (2021); Team (2022a) (a set of short documents translated into 200 languages), which has been extended to FLEURS Conneau et al. (2022) (speech for 102 languages) and Belebele Bandarkar et al. (2023) (reading comprehension questions for 122 languages). Multiway benchmarks admit better comparisons of translation quality across languages because they reduce irrelevant variation in topic or style. Independently constructed benchmarks give a more representative view of real-world performance because they test out-of-domain generalization rather than held-out generalization; fresh queries from users do not necessarily come from the training distribution.

While YouTube-ASL Uthus et al. (2023) takes a first step towards this kind of robust evaluation by reporting zero-shot scores on How2Sign (i.e., scores without finetuning on How2Sign), this test set (like others in sign language translation) is limited in many ways beyond just lack of standardization. For example, the content is translated live after only one look-over, by a combination of Hearing interpreters and Deaf signers (not specified to be professional Deaf interpreters), with errors and redos included in-stream but skipped with sentence-level clip boundaries (complicating discourse-level evaluation) Duarte et al. (2021); Tanzer et al. (2024a). Of course, zero-shot evaluation on a held-out test split also forces researchers to discard valuable training data.

In this paper, we introduce FLEURS-ASL, an extension of FLORES/FLEURS to support their first sign language, American Sign Language. FLEURS-ASL can be used to evaluate a variety of tasks--primarily sentence- and discourse-level translation, but also in principle caption alignment, retrieval, and multiple-choice receptive comprehension--between ASL and all the languages supported by FLORES/FLEURS. FLEURS-ASL was translated by 5 Certified Deaf Interpreters with \(\sim\)5-6 hours of preparation per 1 hour of translated content, ensuring a high quality bar.



We provide several sets of baselines. First, we provide human baselines for sentence-level and discourse-level translation, at 13.0 BLEU (64.6 BLEU) and 13.5 BLEU respectively.

Second, as a model baseline for the suite of FLEURS-ASL tasks, we introduce a unified sign language to text modeling approach that builds upon YouTube-ASL Uthus et al. (2023)'s T5 baseline. Inspired by Whisper's approach for speech to text tasks Radford et al. (2022), we extend the context window to 34 seconds of signing, include 256 tokens of prior text context, incorporate timestamp tokens as both input and output, and train on random video clips. The use of multi-caption random clips heeds the call of Tanzer et al. (2024a) to explore methods that can incorporate more context and are more tolerant to caption misalignment. This exploratory method meets or exceeds the performance of sentence-level baselines (3.7 vs. 2.9 BLEU, 37.2 vs. 33.6 BLEURT) while enabling new tasks through chunked autoregression, such as discourse-level translation, timed translation, and (albeit poorly) caption alignment--despite minimal tuning due to the large design space.

And third, we use FLEURS-ASL to show that recent multimodal frontier models such as GPT-4o, Claude 3 Opus, and Gemini 1.5 Pro display virtually no understanding of ASL, underscoring the importance of including sign language tasks in standard evaluation suites.

We publicly release FLEURS-ASL at this link. We hope that it will encourage development and help to evaluate more generally capable sign language models, perhaps drawing inspiration from our new unified modeling approach. We also hope that the practices and findings from our data collection will aid in the construction of high-quality eval sets for more of the world's sign languages. See Section 6 for a discussion of limitations; while there is great value to standardized benchmarks, we expect that more evaluation will be needed specifically for sign languages to capture variety across vision and language aspects and ensure consistent performance.

## 2 Related Work

### Sign Language Translation Benchmarks

Sign language translation has historically been evaluated on test splits of the same underlying dataset used for training. Early and well-worn datasets like RWTH-PHOENIX-Weather 2014 T Camgoz et al. (2018) have narrow domains--which were initially useful for measuring modeling progress but are not representative of broader content--and consist of live hearing interpretations. Over time datasets have expanded to cover more diverse topics, levels of signing proficiency, recording environment, and so on, Albanie et al. (2021); Hanke et al. (2020); Muller et al. (2022); Gueuwou et al. (2023); Joshi et al. (2023); Muller et al. (2023), but evaluation still tends to be performed on held-out test splits (often even with overlapping signers). This is true for the most related prior work, How2Sign Duarte et al. (2021), the canonical ASL to English translation benchmark. Beyond the nature of the test split, How2Sign has several issues with translation quality (live interpretation with quality that varies significantly between signers) Tanzer et al. (2024a), though it remains valuable due to its unique domain of "how to" instructional videos.

The notable exception to this evaluation trend is YouTube-ASL Uthus et al. (2023) (and the subsequent works that use it Rust et al. (2024); Gueuwou et al. (2024)); YouTube-ASL provides no test split but instead evaluates zero-shot and finetuned results on How2Sign.2 Our goal for FLEURS-ASL is to enhance this kind of evaluation across a wide array of tasks, with a focus on quality over quantity.

Footnote 2: YouTube data is also unsuitable as a benchmark because videos may be deleted over time.

### Machine Translation Benchmarks

In mainstream machine translation, this shift came with benchmarks like FLORES Guzman et al. (2019); Goyal et al. (2021); Team (2022a), an independently constructed, massively multilingual multiway translation benchmark for over 200 languages as text, which has been extended to FLEURS Conneau et al. (2022) for 102 languages as speech, Belebe Bandarkar et al. (2023) with reading comprehension questions for 122 languages, and more through the FLORES+ open source project.

These are by no means the only machine translation benchmarks--there is still space for benchmarks serving other purposes, such as avoiding contamination Bapna et al. (2022), hosting fair WMT competitions in various domains, exercising particular grammatical Bawden et al. (2018) or social aspects Stanovsky et al. (2019); Costa-jussa et al. (2022), and testing extreme new model capabilities Tanzer et al. (2024b), among others--but FLORES/FLEURS serve as a valuable foundation 

for evaluating breadth and robustness of coverage in multilingual models, measured across languages on the same (independent) domain.

### Sign Language Translation Methods

The bulk of work on methods for sign language translation focuses on alternative featurization, architectures, pretraining, or losses for the same underlying sentence-level translation task framing Yin and Read (2020); Moryossef et al. (2021); De Coster et al. (2021); Zhang et al. (2023); Shi et al. (2022); Gueuwou et al. (2024); Hamidullah et al. (2024).

To the best of our knowledge, the only prior work that has trained models for context-aware sign language translation is Sincan et al. (2023), which incorporates prior text or sign spottings as conditioning for sentence-level translation; the training and test examples still rely on sentence-level alignments being available. BOBSL Albanie et al. (2021) and WMT-SLT 23 Muller et al. (2023) include large amounts of weakly supervised data that to our knowledge have not been specially utilized. Tanzer et al. (2024a) argues based on linguistic survey and an empirical case study that isolated sentence-level clips of the kind in How2Sign are often not even comprehensible to human signers; this motivates our new longer-context modeling approach where we train a mixture of tasks, including conditioning on context and generating timed tracks as outputs, on random crops3 of videos with captions that are generally well-aligned but do not strictly contain the corresponding signing.

Footnote 3: Naturally, works that perform self-supervised pretraining on uncaptioned videos already use random clips. Hu et al. (2021)

## 3 The FLEURS-ASL Benchmark

The FLORES benchmark consists of 3001 sentences across 842 articles drawn from English Wikipedia, split into a dev set of 997 sentences (281 articles), devtest set of 1012 sentences (281 articles), and test set of 992 sentences (280 articles). However, they do not release the test set so--like FLEURS--we build on only the dev and devtest sets. We randomly split these sets into halves (4 chunks in total), each intended to be translated by a different interpreter. This strikes a compromise between signer diversity and the feasibility of recruiting/onoarding qualified interpreters.

### Data Collection

We set out specifically to produce translations with a high quality bar, which meant recruiting Certified Deaf Interpreters (CDIs: Dear or Hard of Hearing signers with native or near-native fluency, cultural competence, and professional certification for their interpreting qualifications) who would research and plan translations with substantial preparation time, then perform and record them (with a ratio of 5-6 hours of prep time for each 1 hour of recorded content). Therefore our selection criteria were as follows: a) _required_: CDI, b) _required_: expertise in translation from English text,4 c) _required_: has their own professional recording environment, and d) _prefered_: not prolific on YouTube (to maintain zero-shotness).5

Footnote 4: Much of the work that CDIs do is actually translating from a Hearing interpreter’s ASL into more natural, culturally aware Defar ASLP (particularly facial expressions and body language), so translating the kind of English text found in FLORES is not necessarily within everyone’s area of expertise.

Footnote 5: Unfortunately, it could not be a hard requirement that the interpreter is completely absent from public web data because the pool of qualified interpreters, especially those who are comfortable interpreting content that will be recorded and made public, is limited.

We ran an initial pilot through a sign language interpretation vendor, which resulted in signer #0's chunk. Then we rerandomized the chunks6 and partnered with the Deaf Professional Arts Network (DPAN), a Deaf-led organization that specializes in American Sign Language media, to recruit 4 more interpreters (signers #1-4). Across both of these vendors, candidates were recruited targeted to the above criteria and then screened based on (paid) sample translations they produced for one article. For each candidate that was ultimately selected, several others were considered who either withdrew for logistical reasons or were not the best fit to translate this content.7 At a high level, the 

interpreters were told the project's expectations for the amount of preparation and asked to sign as naturally as possible, flowing naturally between sentences (but preserving the original ordering, which unfortunately limits discourse phenomena to some extent) so that caption alignment wouldn't be artificially easy.8 See Appendix A for more details on the guidance interpreters were given. They then proceeded to translate their content. All interpreters gave informed consent for the use of this data in machine learning and were paid at market rates.

Footnote 8: Even so, there are sometimes unnaturally long pauses between sentences due to the performance/memory limitations of sign language translation, so we expect caption alignment to be less challenging than in real data.

Each interpreter contributed a certain number of hours of work to the task, but there was wide variation in the speed of the interpreters--both in terms of preparation time, and also literally the speed of their signing. For the faster interpreters (#1 and #3), we went through a round of revisions where the first author (a non-native but proficient signer) quality-checked all the content and sent feedback to DPAN for rerecordings--minor performance errors like unintentionally omitting details, not high-level changes to the translations, which were of course up to the discretion of the expert interpreters.9 Given time constraints, for the other interpreters (#2 and #4) we decided to target fewer hours of translated content and do fewer revisions. This means that signer #1 and #3 have the most complete and thoroughly revised content, but the others still meet a high quality bar and offer more variation for evaluation. The first author manually annotated caption alignments for all content while performing quality checks.

Footnote 9: This is another way in which high-quality sign language translation is more difficult than translation for text: the translations are performed in real-time (often based on nonstandardized notes) and cannot be edited later. If there is an error (and if we care about discourse-level tasks, which we do here), and it would be interesting to understand how much of these matters can be understood.

### Dataset Statistics

See Table 1 for the final statistics for the translated content in FLEURS-ASL, broken down per signer. Of the 5 signers, 2 are men and 3 are women. Signers #0-3 are right-handed, and #4 is left-handed. There is limited variation in skin tone and age; see discussion in Section 6. Note the variation in median sentence duration across signers (from 10.3 seconds to 22.7 seconds) despite minimal variation in median sentence length (i.e., signing speed varies across interpreters), and the increased median sentence duration for FLEURSASL vs. How2Sign (13.2 seconds vs. 5.3 seconds) due to the different domain (web articles vs. informal instructional narratives). The short duration of How2Sign clips (especially in the left tail of the distribution) can make them difficult to understand for sentence-level translation in the absence of con

Figure 1: **FLEURS-ASL dataset splits.** The sentences are divided among 5 interpreters, and 3 sets of splits: zero-shot (“zs”), signer-independent finetuning (“si”), and signer-dependent finetuning (“sd”). We blur the interpreters’ faces in this paper for privacy, but the underlying dataset is unblurred because facial expressions are an essential component of the grammar of sign languages.

 

text (Tanzer et al., 2024a). See Appendix C for more description of the individual signers.

### Benchmark Tasks

See Figure 1 for a depiction of the final dataset splits. We provide splits for three settings: zero-shot, signer-independent finetuning (i.e., no signers overlap between the train and test sets), and signer-dependent finetuning (i.e., all signers overlap).10 We expect that the signer-dependent splits will be used less than the others, but they may be useful for studying the effects of per-signer personalization.

Footnote 10: We ensure that the signer-dependent splits have disjoint train and test sets in light of the content overlap for signers #0 and #1+#2.

FLEURS-ASL supports many tasks by virtue of being grafted onto FLORES/FLEURS/Belebele, but we expect that the most common will be sentence- and discourse-level translation from ASL to English. We recommend scoring sentence-level translation with both BLEU Papineni et al. (2002) (with sacreBLEU Post (2018) with intl tokenization) and BLEURT Sellam et al. (2020) (with BLEUT-20 Pu et al. (2021)). Untimed discourse-level translation can only be scored with BLEU because BLEURT is not intended for long outputs. We score timed translation using Timed BLEUCherry et al. (2021) (and by extension, Timed BLEURT), where we temporally interpolate at the granularity of characters rather than tokens for convenience.

## 4 Baselines

We provide three sets of baselines. First, we give human baselines for sentence- and discourse-level translation. Second, we train new versions of the YouTube-ASL baselines that can support many FLEURS-ASL sign\(\rightarrow\)text tasks using a unified multitask mixture, integrating prior text history, timestamps, and a longer context window (rather than just caption-level translation). And third, we evaluate current multimodal frontier models and show that they have virtually no understanding of ASL.

### Human Baseline

We recruited a native Deaf ASL signer with professional training/experience in sign language education through DPAN to perform the human baseline. The annotator was informed of the purpose of their translations (to be released for use in ML research, to understand how well models perform at ASL\(\rightarrow\)English translation vs. humans) and paid at market rates.

We sampled 8 articles uniformly at random from the signer-dependent test set for each signer. Within each article, we sampled one sentence. For each article, the annotator was tasked with translating that sampled sentence's ASL clip back to English, and then (once finished) was presented with the entire article's ASL video and asked to translate it to English in its entirety. This gives us human baselines for sentence- and discourse-level translation across 40 instances each, where like Tanzer et al. (2024a) the translations for each setting are performed by the same person (to reduce irrelevant variation) without contaminating the sentence-level translations with discourse context. See Appendix B for the task instructions.

The scores are 13.0 BLEU (64.6 BLEURT) for sentence-level translation and 13.5 BLEU for discourse-level translation. Like in Tanzer et al. (2024a), these metrics (in particular BLEU) may sound low, but qualitatively the human translations look good. The discrepancies with the references

\begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline  & \# & \# & \# & sent. length (chars) & sent. duration (secs) & \# \\  & signers & discourses & sentences & (0, 10, 50, 90, 100)\% & (0, 10, 50, 90, 100)\% & hours \\ \hline How2Sign & 11 & 2456 & 35263 & 3, 27, 76, 171, 977 & 0.0, 1.7, 5.3, 13.6, 143.0 & 79.1 \\ \hline FLEURS-ASL & 5 & 495 & 1749 & 28, 76, 122, 186, 368 & 1.6, 7.4, 13.3, 25.0, 72.8 & 7.49 \\ _signers_ & & & & & & \\ \#0 & 1 & 103 & 350 & 35, 71, 122, 190, 328 & 4.4, 8.1, 15.1, 24.4, 48.5 & 1.64 \\ \#1 & 1 & 141 & 508 & 35, 81, 123, 183, 368 & 1.6, 6.9, 12.0, 19.6, 46.0 & 1.84 \\ \#2 & 1 & 70 & 252 & 52, 75, 124, 190, 328 & 4.4, 12.6, 22.7, 37.0, 72.8 & 1.72 \\ \#3 & 1 & 141 & 494 & 28, 74, 123, 184, 290 & 2.4, 6.6, 10.3, 15.4, 30.5 & 1.57 \\ \#4 & 1 & 40 & 145 & 33, 80, 117, 165, 246 & 4.3, 8.6, 15.5, 25.6, 38.5 & 0.72 \\ \hline \hline \end{tabular}
\end{table}
Table 1: **FLEURS-ASI summary statistics** and comparisons to How2Sign (all splits included). # of signers, # of discourses, # of sentences, sentence length & duration (across 0th, 10th, 50th, 90th, and 100th percentiles) measured in characters and seconds respectively, and # of hours.

 

come from a variety of sources (natural changes due to backtranslation through a sign language, errors in one of the translation directions) but the most interesting come from visual ambiguity in the sign language translation, such as when "6,500" was perceived as "2,500" because the 6 was angled such that the ring and middle finger were aligned, making it look like only 2 fingers were raised without close inspection. See Appendix E for the full set of human translations.

### Unified Sign to Text Modeling

#### 4.2.1 Method

Inspired by Whisper [7] and heeding the call of Tanzer et al. (2024a) to move beyond sentence-level sign language translation, we present a new unified modeling approach to flexibly support many FLEURS-ASL tasks (and more) in one model. We integrate timestamp tokens and previous text tokens with a 34-second context window trained on random clips from captioned videos. This approach fits into the typical encoder-decoder framework without architectural modifications, only changes to data preparation. See Figure 2 for a complete description of this procedure. The key point is that, in contrast to prior work in sign language, our training examples contain timed caption _tracks_ as supervision rather than individual phrase- or sentence-level captions, and the timing of these tracks is used to tolerate random video clip boundaries, rather than determine the boundaries of those clips. This disregard for

Figure 2: **Unified multitask document-level sign to text training; see Figure 3 in Appendix D for a complete exposition. Our baselines train on YouTube-ASL and evaluate on FLEURS-ASL.**



preordained alignments in our clip sampling enables us to handle arbitrary clipping at inference time: in particular, we can perform transformations of long videos in autoregressive chunks, where the next chunk starts after the previous chunk's predicted timestamps.11

Footnote 11: Specifically: in addition to providing caption history in the input text tokens, we maintain the first 4 seconds and last 10 seconds of the 34-second context window as regions where predicted captions should not start or stop; if a caption is predicted in that range, we shift the window for autoregression to ensure that the model has more context to perform its translation, and to tolerate misalignment. This heuristic was determined interactively by observing when the autoregression got “stuck” on certain videos (not from the test set, which would cause leakage).

#### 4.2.2 Experiments

We build off YouTube-ASL [86]'s baseline, which scales 85 3D points from MediaPipe Holistic [97, 10] skeletons to fit in a unit cube across time, then linearly projects them (at half frame rate) into the encoder of T5 [11], and predicts text tokens from the decoder (using beam search with beam size 5). (T5 is an encoder-decoder transformer pretrained on English text.) We compare to a fresh baseline for sentence-level translation so that results are meaningful in light of dataset churn, and report results on context+sentence-level translation (sentence-level translation given prior text history) and discourse-level translation as well. We train each model on 64 TPUv3s for 200k steps with batch size 128 and Adafactor [12] learning rate 0.001 using the T5X framework [135] (10k steps = \(\sim\)1 hour), with validation BLEU as the checkpoint selection criterion. We finetune on How2Sign for at most 10k steps and FLEURS-ASL for at most 100 steps. We first train the model on caption-level data, as in YouTube-ASL, for 100k steps, then mix in the multitask mixture with weight 0.5.12 See Tables 2, 3, and 4 for quantitative results; see Appendix C for expanded experimental results disaggregated for each individual signer, following Tanzer et al. (2024a); and see Table 6 for qualitative examples.

Footnote 12: This could be seen as a kind of context window extension training [135, 136].

The highlights are that our method performs at least as well as the caption-level baseline at sentence-level translation (3.7 vs. 2.9 BLEU, 37.2 vs. 33.6 BLEURT) while being able to tackle additional tasks. The models improve with signer-independent and then signer-dependent finetuning (but much less than the gap on How2Sign), as well as with additional context.

Results are still far below human performance of 13.0 BLEU (64.6 BLEURT). This approach also shows poor performance on caption alignment--far below a strong model-free baseline where caption duration is scaled in proportion to caption length--because the model can decohere from the original input (especially when captions are long relative to the context window) as the model autoregresses. This is not an issue for translation because it does not have to follow along with a given input track.

\begin{table}
\begin{tabular}{l|c|c} \hline \hline  & **How2Sign** & **FLEURS-ASL** \\  & sentence-level & sentence-level & context+sentence-level \\ \hline Caption-level & 4.0 (34.4) / / 12.1 (45.4) & 2.9 (33.6) / 4.8 (38.6) / 5.4 (40.3) & - \\ Ours & 5.1 (36.9) / - 14.4 (46.3) & 3.7 (37.2) / 4.9 (39.7) / 6.3 (42.2) & 3.9 (38.3) / 5.3 (41.4) / 6.8 (43.4) \\ \hline Human & \(\sim\) 19.8 (56.6) & \(\sim\) 13.0 (64.6) & - \\ \hline \hline \end{tabular}
\end{table}
Table 2: **Baseline results for sentence-level translation from ASL to English (zero-shot / signer-independent finetuning / signer-dependent finetuning), measured in BLEU (\(\uparrow\)) (BLEUrt in parentheses). We also report scores on How2Sign as a point of reference. Note that the human baseline scores are not perfectly comparable for either benchmark because they are for a subset of the data that is not necessarily representative due to the sampling method and small size. Human baseline for How2Sign is from Tanzer et al. (2024a).**

\begin{table}
\begin{tabular}{l|c c} \hline \hline  & \multicolumn{2}{c}{**discourse-level**} \\  & untimed & timed \\ \hline Ours & 4.0 (-) & 3.1 (31.7) \\ \hline Human & 13.5 (-) & - \\ \hline \hline \end{tabular}
\end{table}
Table 3: **Baseline results for discourse-level translation from ASL to English (untimed vs. timed). Untimed is measured in BLEU (\(\uparrow\)), because BLEURT is not intended for long outputs; timed is measured in timed BLEU (timed BLEUt in parentheses) [135]. We perform the temporal interpolation for timed BLEU/BLEUrt at the character level.** 



### Frontier Models

Finally, we evaluate 3 multimodal frontier models--Claude 3 Opus Anthropic (2024), GPT-4o OpenAI (2024), and Gemini 1.5 Pro Team (2024)--on the same FLEURS-ASL sample we used for the human baseline. While Claude 3 Opus only claims support for image inputs (but accepts up to 20 image inputs), GPT-4o and Gemini 1.5 Pro support video, at least at low frame rates. We prompt the model to "Translate the following video from American Sign Language to English. Give only the translation without any extra explanations." and upload the given video at 1 Hz, sampling with default parameters. See quantitative results in Table 6 and qualitative results in Table 6. None of the models show any appreciable sign language understanding. Claude 3 sometimes acknowledges that it cannot understand the videos and rejects the query.

Overselling technological progress is all too common in the sign language space and is a point of frustration for the Deaf community. In the brief time that these models have been out, we have personally witnessed multiple independent instances of users testing sign language examples and then declaring that the models must understand sign language, when the examples are either trivial/given away by context or the translations are completely incorrect. We hope that FLEURS-ASL will become a standard addition to frontier model evaluation suites to increase the visibility of sign language tasks and calibrate expectations for them.

## 5 Conclusion

In this paper, we presented FLEURS-ASL, an extension of the FLORES/FLEURS translation benchmarks to their first sign language, American Sign Language. FLEURS-ASL distinguishes itself from prior sign language benchmarks not just with its flexibility (supporting multiple tasks and translation directions by virtue of extending a larger benchmark) but also its domain-specific quality considerations (use of Certified Deaf Interpreters translating English text with substantial preparation time, rather than semi-live interpretation following a video). We provided human baselines, domain-specific model baselines (using a new unified multitask approach that operates on random clips), and multimodal frontier model baselines; these collectively show that there is still a long way to go to reach usable performance on ASL to English translation. We hope that FLEURS-ASL will encourage more focus on this task--framed in a way that deemphasizes strictly aligned captions and therefore admits use of noisier datasets--and serve as scaffolding to extend FLORES/FLEURS to even more of the world's sign languages.

## 6 Limitations

While FLEURS-ASL improves upon the quality and flexibility of prior evaluations for sign language understanding, it still has a number of limitations, both in language and vision aspects.

In terms of language: The FLORES Guzman et al. (2019) sentences were originally drawn from Wikipedia and therefore have a limited domain, favoring third-person over first- and second-person (in monologues only; no dialogues) and lacking Deaf-centric vocabulary and topics. This content was translated from English into ASL, which means that (despite efforts to maintain a high quality bar) the content will surely differ linguistically in some respects from spontaneously uttered ASL. As described in Section 3.1, in order to support sentence-level tasks and caption alignment, we imposed restrictions on reordering of content across sentences that limit discourse structure, and the translation guidelines were tailored primarily to elicit data suitable for evaluating sign language understanding rather than generation.13 FLEURS

\begin{table}
\begin{tabular}{l l} \hline \hline  & frame-acc \\ \hline Length scaling & 88.5\% \\ Ours & 24.3\% \\ \hline \hline \end{tabular}
\end{table}
Table 4: **Baseline results for ASL-English caption alignment**, measured in frame accuracy (\(\uparrow\)) as in Albanie et al. (2021); Bull et al. (2021). “Length scaling” refers to a model-free baseline where the caption boundaries are linearly interpolated along the video duration according to the length (in characters) of the corresponding text. Our model often decoheres from the input track while autoregressive and therefore drastically underperforms the length rescaling baseline.

\begin{table}
\begin{tabular}{l c c} \hline \hline  & BLEU & BLEURT \\ \hline GPT-4o & 0.1 & 32.2 \\ Claude 3 Opus & 0.4 & 32.0 \\ Gemini 1.5 Pro & 0.2 & 20.6 \\ \hline Human & 13.0 & 64.6 \\ \hline \hline \end{tabular}
\end{table}
Table 5: **Frontier multimodal models on sentence-level translation from ASL to English.** None of the models demonstrate meaningful understanding of ASL; the variation in scores reflects artifacts of output length.



ASL currently lacks finer-grained annotations than sentence alignments; we hope that the community could contribute these over time if useful. In terms of representativeness, variation in regional dialect and signing style (e.g. due to age or environment where ASL was acquired) is limited due to the use of only 5 signers--signers with the background to become CDIs. In particular, many Deaf/Hard of Hearing people are not exposed to ASL in early childhood, and sign language translation systems should still work for them. And of course FLEURS-ASL only covers American Sign Language, one of the >300 sign languages used worldwide.

Footnote 18: [https://www.facebook.com/en/2019/04/](https://www.facebook.com/en/2019/04/)

In terms of vision: The FLEURS-ASL translations are performed in professional recording environments with clean backgrounds, which means that issues relating to camera quality/perspective or environment complexity are not captured by the benchmark. Our interpreters do not cover all phenotypic variation (e.g., skin tone), so the benchmark cannot guarantee fairness across demographics; this is impossible in a set of only 5 signers anyway. We circumvent this problem somewhat in our baselines by using MediaPipe Holistic, which has its own fairness evaluations in its model cards (med), but the benchmark is intended to be method-agnostic. Many variations in physical appearance correspond with sociolects (e.g., race, gender, and age Ceil Lucas and Bayley [2001]), so this is not strictly a vision problem. Note that modeling poses of the kind produced by MediaPipe Holistic is also a fundamentally flawed approach in general because it does not capture semantically relevant features like the tongue (and from the perspective of fairness, cannot handle people with atypically numbered or formed fingers and hands).

Additionally, in terms of truly zero-shot evaluation: Because FLORES/FLEURS is seeded from Wikipedia and widely used for finetuning or evaluation, the text/speech references for FLEURS-ASL may be contaminated in pretraining data. The FLEURS-ASL interpreters may also appear in public data, such as on YouTube, due to the limited pool of suitably qualified candidates.

## Ethics Statement

The interpreters who translated FLEURS-ASL were paid at market rates and consented to their video translations being published for use in machine learning research by the broader community. While facial expressions are essential to sign language grammar and therefore must be included in the dataset, we ask that you blur the interpreters' faces when including examples in publications (as we do in this paper). You should not attempt to reidentify the interpreters or use their likenesses to generate and publish other content (deepfakes). Please be culturally respectful of the Deaf/Hard of Hearing community in your use of the benchmark and do not exaggerate the broader significance of advances on it.

## References

* Albanie et al. (2021) Samuel Albanie, Gul Varol, Liliane Momeni, Hannah Bull, Triantafyllos Afouras, Himel Chowdhury, Neil Fox, Bencie Woll, Rob Cooper, Andrew McParland, and Andrew Zisserman. 2021. BBC-oxford british sign language dataset. arXiv preprint. Cited by: SS1.
* Anthropic (2024) The claude 3 model family: opus, sonnet, haiku. Cited by: SS1.
* L. Bandarkar, D. Liang, B. Muller, M. Artetxe, S. Narayan Shukla, D. Husa, N. Goyal, A. Krishnan, L. Zettlemoyer, and M. Khalsa (2023) The belbele benchmark: a parallel reading comprehension dataset in 122 language variants. Preprint. External Links: 2308.16884 Cited by: SS1.
* Ankur Bapna, I. Caswell, J. Kreutzer, O. Firat, D. van Esch, A. Siddhant, M. Niu, P. Baljekar, X. Garcia, W. Riesa, Y. Cao, M. Xu Chen, K. Macherey, M. Krikun, P. Wang, A. Gutkin, A. Shah, Y. Huang, Z. Chen, Y. Wu, and M. Hughes (2022) Building machine translation systems for the next thousand languages. Preprint. External Links: 2205.03983 Cited by: SS1.
* R. Bauken, R. Sennrich, A. Birch, and B. Haddow (2018) Evaluating discourse phenomena in neural machine translation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), New Orleans, Louisiana, pp. 1304-1313. External Links: Link Cited by: SS1.
* H. Bull, T. Afouras, G. Varol, S. Albanie, L. Momeni, and A. Zisserman (2021) Aligning subtitles in sign language videos. Preprint. External Links: 2105.02877 Cited by: SS1.

 

Necati Cihan Camgoz, Simon Hadfield, Oscar Koller, Hermann Ney, and Richard Bowden. 2018. Neural sign language translation. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
* Lucas and Bayley [2001] Clayton Valli Ceil Lucas and Robert Bayley. 2001. _Sociolinguistic Variation in American Sign Language_. Gallaudet University Press.
* Chen et al. [2023] Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. 2023. Extending context window of large language models via positional interpolation. _Preprint_, arXiv:2306.15595.
* Cherry et al. [2021] Colin Cherry, Naveen Arivazhagan, Dirk Padfield, and Maxim Krikun. 2021. Subtitle Translation as Markup Translation. In _Proc. Interspeech 2021_, pages 2237-2241.
* Conneau et al. [2022] Alexis Conneau, Min Ma, Simran Khanuja, Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna. 2022. Fleurs: Few-shot learning evaluation of universal representations of speech. _Preprint_, arXiv:2205.12446.
* Costa-jussa et al. [2022] Marta R. Costa-jussa, Christine Basta, and Gerard I. Gallego. 2022. Evaluating gender bias in speech translation. _Preprint_, arXiv:2010.14465.
* De Coster et al. [2021] Mathieu De Coster, Karel D'Oosterlinck, Marija Pizurica, Paloma Rabaey, Severine Verlinde, Mieke Van Herreweghe, and Joni Dambre. 2021. Frozen pretrained transformers for neural sign language translation. In _Proceedings of the 1st International Workshop on Automatic Translation for Signed and Spoken Languages (AT4SSL)_, pages 88-97, Virtual. Association for Machine Translation in the Americas.
* Duarte et al. [2021] Amanda Duarte, Shruti Palaskar, Lucas Ventura, Deepti Ghadiyaram, Kenneth DeHaan, Florian Metze, Jordi Torres, and Xavier Giro-i Nieto. 2021. How2Sign: A Large-scale Multimodal Dataset for Continuous American Sign Language. In _Conference on Computer Vision and Pattern Recognition (CVPR)_.
* Goyal et al. [2021] Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc'Aureliio Ranzato, Francisco Guzman, and Angela Fan. 2021. The flores-101 evaluation benchmark for low-resource and multilingual machine translation. _Preprint_, arXiv:2106.03193.
* simultaneous face, hand and pose prediction, on device.
* Guerwoou et al. [2024] Shester Guerwoou, Xiaodan Du, Greg Shakhnarovich, and Karen Livescu. 2024. Signmusketeers: An efficient multi-stream approach for sign language translation at scale. _Preprint_, arXiv:2406.06907.
* Gueuwou et al. [2023]Shester Gueuwou, Sophie Siake, Colin Leong, and Mathias Muller. 2023. JWsign: A highly multilingual corpus of bible translations for more diversity in sign language processing. _Preprint_, arXiv:2311.10174.
* Guzman et al. [2019] Francisco Guzman, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, and Marc'Aurelio Ranzato. 2019. Two new evaluation datasets for low-resource machine translation: Nepali-english and sinhala-english.
* Hamidullah et al. [2024] Yasser Hamidullah, Josef van Genabith, and Cristina Espana-Bonet. 2024. Sign language translation with sentence embedding supervision. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 425-434, Bangkok, Thailand. Association for Computational Linguistics.
* Hanke et al. [2020] Thomas Hanke, Marc Schulder, Reiner Konrad, and Elena Ahn. 2020. Extending the Public DGS Corpus in size and depth. In _Proceedings of the LREC2020 9th Workshop on the Representation and Processing of Sign Languages: Sign Language Resources in the Service of the Language Community, Technological Challenges and Application Perspectives_, pages 75-82, Marseille, France. European Language Resources Association (ELRA).
* Hu et al. [2021] Hezhen Hu, Weichao Zhao, Wengang Zhou, Yuechen Wang, and Houqiang Li. 2021. Signbert: Pre-training of hand-model-aware representation for sign language recognition. _Preprint_, arXiv:2110.05382.
* Joshi et al. [2023] Abhinav Joshi, Susmit Agrawal, and Ashutosh Modi. 2023. lstranslate: Dataset for translating indian sign language. _Preprint_, arXiv:2307.05440.
* Logaresci et al. [2019] Camillo Logaresci, Jiuqiang Tang, H Radon Nash, Chris McClahan, Esha Ubowveja, Michael Hays, Fan Zhang, Chuo-Ling Chang, Ming Guang Yong, Juhyun Lee, Wan-Teh Chang, Wei Hua, Manfred Georg, and Matthias Grundmann. 2019. MediaPipe: A framework for building perception pipelines. _Preprint_, arXiv:1906.08172.
* Morysosef et al. [2021] Amit Morysosef, Ioannis Tsochantaridis, Joe Dinn, Necati Cihan Campoz, Richard Bowden, Tao Jiang, Annette Rios, Mathias Muller, and Sarah Ebling. 2021. Evaluating the immediate applicability of pose estimation for sign language recognition. _Preprint_, arXiv:2104.10166.
* Muller et al. [2023] Mathias Muller, Malihe Alikhani, Eleftherios Avramidis, et al. 2023. Findings of the second WMT shared task on sign language translation (WMT-SLT23). In _Proceedings of the Eighth Conference on Machine Translation_, pages 68-94, Singapore. Association for Computational Linguistics.
* Muller et al. [2020] Mathias Muller, Sarah Ebling, Eleftherios Avramidis, Alessia Battisti, Michele Berger, Richard Bowden, 

Annelies Braffort, Necati Cihan Camgoz, Cristina Espana-bonet, Roman Grundkiewicz, Zifan Jiang, Oscar Koller, Amit Moryossef, Regula Perrollaz, Sabine Reinhardt, Annette Rios, Dimitar Stierionov, Sandra Sidler-miserez, and Katja Tissi. 2022. Findings of the first WMT shared task on sign language translation (WMT-SLT22). In _Proceedings of the Seventh Conference on Machine Translation (WMT)_, pages 744-772, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
* OpenAI [2024] OpenAI. 2024. Gpt-4o system card. _Preprint_, arXiv:2410.21276.
* Papineni et al. [2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
* Peng et al. [2023] Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2023. Yarn: Efficient context window extension of large language models. _Preprint_, arXiv:2309.00071.
* Post [2018] Matt Post. 2018. A call for clarity in reporting BLEU scores. In _Proceedings of the Third Conference on Machine Translation: Research Papers_, pages 186-191, Belgium, Brussels. Association for Computational Linguistics.
* Pu et al. [2021] Amy Pu, Hyung Won Chung, Ankur P Parikh, Sebastian Gehrmann, and Thibault Sellam. 2021. Learning compact metrics for mt. In _Proceedings of EMNLP_.
* Radford et al. [2022] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2022. Robust speech recognition via large-scale weak supervision. _Preprint_, arXiv:2212.04356.
* Raffel et al. [2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. _Preprint_, arXiv:1910.10683.
* Rusat et al. [2024] Phillip Rust, Bowen Shi, Skyler Wang, Necati Cihan Campoz, and Jean Maillard. 2024. Towards privacy-aware sign language translation at scale. _Preprint_, arXiv:2402.09611.
* Sellam et al. [2020] Thibault Sellam, Dipanjan Das, and Ankur P. Parikh. 2020. Bleurt: Learning robust metrics for text generation. _Preprint_, arXiv:2004.04696.
* Shazeer and Stern [2018] Noam Shazeer and Mitchell Stern. 2018. Adafactor: Adaptive learning rates with sublinear memory cost. _Preprint_, arXiv:1804.04235.
* Shi et al. [2022] Bowen Shi, Diane Brentari, Greg Shakhnarovich, and Karen Livescu. 2022. Open-domain sign language translation learned from online video. _arXiv preprint_.
* Mencanolgu et al. [2023] Ozge Mercanoglu Sincan, Necati Chian Camgoz, and Richard Bowden. 2023. Is context all you need? scaling neural sign language translation to large domains of discourse. _Preprint_, arXiv:2308.09622.
* Stanovsky et al. [2019] Gabriel Stanovsky, Noah A. Smith, and Luke Zettlemoyer. 2019. Evaluating gender bias in machine translation. _Preprint_, arXiv:1906.00591.
* Tanzer et al. [2024a] Garrett Tanzer, Maximus Shengelia, Ken Harrenstein, and David Uthus. 2024a. Reconsidering sentence-level sign language translation. _Preprint_, arXiv:2406.11049.
* Tancer et al. [2024b] Garrett Tanzer, Mirac Suzgun, Eline Visser, Dan Jurafsky, and Luke Melas-Kyriazi. 2024b. A benchmark for learning to translate a new language from one grammar book. _Preprint_, arXiv:2309.16575.
* Team [2024] Gemini Team. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. _Preprint_, arXiv:2403.05530.
* Team [2022a] NLLB Team. 2022a. No language left behind: Scaling human-centered machine translation.
* Team [2022b] TSX Team. 2022b. Scaling up models and data with t5x and seqio. _arXiv preprint arXiv:2203.17189_.
* Uthus et al. [2023] David Uthus, Garrett Tanzer, and Manfred Georg. 2023. Youtube-asl: A large-scale, open-domain american sign language-english parallel corpus. _Preprint_, arXiv:2306.15162.
* Yin et al. [2021] Kayo Yin, Amit Moryossef, Julie Hochgesang, Yoav Goldberg, and Malihane Alikhani. 2021. Including signed languages in natural language processing. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 7347-7360, Online. Association for Computational Linguistics.
* Yin and Read [2020] Kayo Yin and Jesse Read. 2020. Better sign language translation with stmc-transformer. _Preprint_, arXiv:2004.00588.
* Zhang et al. [2023] Biao Zhang, Mathias Muller, and Rico Sennrich. 2023. Sltunet: A simple unified model for sign language translation. _Preprint_, arXiv:2305.01778.

 

Translation Guidance

While sign language interpreting and translation is a mature field with comprehensive training and certification processes, the desiderata for a machine translation benchmark are somewhat unique compared to the variety of contexts in which interpreting and translation are typically performed. We gave the following guidance to the interpreters to consistently elicit the kind of content we were looking for (not necessarily verbatim; some guidance was provided live during onboarding or as feedback).

* This is a challenging set of content to translate, with relatively short narratives about many different topics. We expect that you will spend 5-6 hours preparing the translations for every 1 hour of recorded content to ensure a high quality standard.
* Try to sign as naturally as possible. We are looking for content that looks like it was originally signed in ASL rather than translated, as much as possible. For example, this means that you don't need to add explanations of concepts or entities that aren't present in the source content, like you might for a translation of a public service announcement. You should sign as you would in a professional environment to another signer with similar background to you.
* Implied by this, do not leave intentional pauses between sentences.
* However, you should not reorder content between sentences. i.e., You should be able to take the original sentences and align them to your translation as a caption track, and the captions should be a reasonable translation of the signed content.
* If there is any ambiguity in the English sentence when you're translating into ASL, act as if the information was already known from context. For example, if the sentence starts with "he", you should not fingerspell "he" to clarify the gender, but instead just point and assume that gender is known from context.
* If there is anything that needs to be specified when translating into ASL, pick one interpretation and stick with it throughout the narrative. For example, if the sentence is describing a "crash", without specifying what kind of vehicle crashed (car, plane, etc.), pick the most reasonable interpretation given the context and go with it.

## Appendix B Human Baseline Instructions

Your task today will be to perform ASL\(\rightarrow\)English translation tasks on a sample of 40 videos we provide to you. We will use/release these translations so that we and the broader AI research community can compare how well AI translates ASL vs. a human (and get a better understanding of its limitations).

**You're allowed to watch the video multiple times, slow down the video, etc. to help you.** But don't agonize too much over it.

**You must perform the tasks in a particular order, in order to ensure the validity of the results.** Specifically, we will ask you to:

1. Look at a clip containing 1 sentence, and translate that clip.
2. Look at a full video containing that sentence plus others around it, and translate the entire video given the additional context.

First, we'll go through an example.

a) Watch the following clip. The task is to translate the clip. My translation of the clip is:

_If you stand at the edge of the water, you'll look down to the bottom and see pebbles and gunk._

b) Watch the following clip. The task is to translate the entire video. (Note that in light of additional context, you may refine your translation from step a for step b.) My translation of the video:

_It will behave similar to water. It's transparent like water. If you stand at the shore and look down, you'll see through it to the pebbles and gunk on the bottom. Stofan adds, "As far as we know, only one planet is more active than Titan: the planet Earth."_

## Appendix C Disaggregated Analysis

First, some subjective observations about the different signers' styles:

Signers #0, #3, and #4 tend to sign closer to the original English syntax when there is no strong reason not to, whereas signers #1 and #2 more freely structure the sentences. Some individual points of distinction 

* Signer #0 performs a swish to separate individual words in a fingerspelled phrase, which is uncommon.
* Signer #1 tends to use more rare signs (such as cities/countries) and vanguard language practices (such as using the endogenous BSL sign for England rather than the traditional ASL sign).
* Signer #2 signs the slowest.
* Signer #3 signs the fastest.
* Signer #4 signs primarily left-handed, but is more ambidextrous than the others.

Second, see Table 7 for disaggregated sentence-level ASL to English translation results. As expected due to our more stringent set of requirements for recruiting interpreters, the human baseline scores (especially BLEURT) in FLEURS-ASL are more consistent across signers than the scores on How2Sign provided by Tanzer et al. (2024a). The BLEURT scores are also higher overall; we speculate that this is due factors such as the source sentences being longer, the translations into ASL being higher quality/more faithful, and the reference domain being more neutral. The variation in scores that remains in FLEURS-ASL does not seem to reflect underlying translation quality, but just the depth of the translation (and e.g., amount of fingerspelling).

## Appendix D Details of Example Format

See Figure 3 for an exposition of the complete task format, expanded from Figure 2.

## Appendix E Complete Human Baseline

See Table 8 for the complete set of FLEURS-ASL human baseline translations, also available at this link.

\begin{table}
\begin{tabular}{c|c} \hline \hline  & sentence-level \\ \hline Caption-level & \\ \#0 & 3.6 (34.8) \\ \#1 & 2.0 (31.8) \\ \#2 & 3.6 (35.7) \\ \#3 & 4.2 (36.2) \\ \#4 & 3.9 (33.8) \\ \hline Ours & \\ \#0 & 5.1 (39.1) \\ \#1 & 2.4 (35.1) \\ \#2 & 4.3 (38.8) \\ \#3 & 5.3 (39.5) \\ \#4 & 4.7 (39.1) \\ \hline Human & \\ \#0 & 12.1 (63.4) \\ \#1 & 9.0 (60.6) \\ \#2 & 6.5 (63.1) \\ \#3 & 16.1 (68.5) \\ \#4 & 21.7 (67.6) \\ \hline \hline \end{tabular}
\end{table}
Table 7: **Results for zero-shot sentence-level translation from ASL to English disaggregated by signer**, measured in BLEU (BLEURT in parentheses).

 

_Top left:_**Model architecture. Input text tokens (token embeddings) and up to 512 frames of half-frame-rate linearly projected MediaPipe Holistic landmarks (subset of 85 3D points) are the inputs to T5V1.1-Base (a pretrained encoder-decoder Transformer), finetuned on YouTube-ASL as described below.

_Right:_**Preprocessing steps to sample training clips from long videos. In order to sample training clips uniformly in proportion to video duration, we perform the following steps. (This is only necessary because SeqIO Team, 2022b) does not support sampling training examples in proportion to scalar values attached to them, and because we want to support multi-epoch training with random crops without duplicating the underlying data many times.) We chunk arbitrary length captioned training videos into training examples on disk of \(2n\) seconds strided by \(n\) seconds, preprocess them with MediaPipe Holistic, and carry along the timed caption track, including the previous and next \(n\) seconds of captions. (Here, \(n\) is 34 seconds so that 15 Hz input fits in 512 tokens.) For each training example, the start position of the clip is sampled from the first half of the chunk uniformly at random, and with probability 0.2 the duration of the clip is truncated to between \(m\) and \(n\) seconds. (Here, \(m\) is 17 seconds). We modify this scheme slightly at the beginning and end of videos; by default, the first and last \(n\) seconds of the video would be extremely undersampled because they do not overlap with other examples. We make the chunk duration \(\frac{3}{2}n\) seconds rather than \(2n\) and sample the start position \(\sim max(0,U[-\frac{n}{2},n])\) for the first chunk and \(\sim min(\frac{n}{2},U[0,\frac{3n}{2}])\) for the last chunk. For videos whose duration doesn't divide evenly by \(2n\) seconds, we overlap the final two chunks so they are between \(\frac{3n}{2}\) and \(2n\) seconds long.

_Bottom left:_**Control token format and mixture.** For implementation convenience, we instantiate control tokens as regular text. In order to save compute, task mixture weights are chosen by intuition without hyperparameter sweeps; we expect that optimal weights would change depending on dataset size and application priorities anyway.

Captions are represented as text spans delimited by separator tokens (either ' ' or '\(\backslash n\)'), optionally with start and end timestamps prepended. There are three sets of captions: the "curr" captions fully contained within the input clip, the "prev" captions starting in the \(n\) seconds prior to the input clip, and the "next" captions ending in the \(n\) seconds after it. There are two main branches of input tokens: caption alignment (with probability 0.04) and translation (0.96).

The caption alignment branch supports two modes, "choose sep" (0.5) where the input specifies caption breaks, and "copy sep" (0.5), where the model predicts them. In the latter case, the average caption duration across the source video can be provided as conditioning (0.5) as a form of controllability. Then the "left edge" caption timestamp is provided, i.e. the end timestamp of any caption that crosses the left boundary of the landmark input, so that the model knows where to start predicting outputs. Finally, two more modes for either "overflow" (0.8), where the provided captions to align may spill past the right boundary of the clip, and "subset" (0.2), where the captions are a random prefix of those covering the clip, in order to support whole-video and human-in-the-loop caption alignment respectively. The target tokens are unconditionally timed, newline-separated captions.

The translation branch supports untimed (0.2) and timed (0.8) translation (determining whether the target captions include timestamp tokens), with the latter also allowing duration conditioning (0.5). Either no captions (0.2), previous captions (0.64), or previous and next captions (0.16) are provided as context, to support isolated translation, blockwise autoregressive translation, and infilling.

Figure 3: **Unified multitask document-level sign to text training, expanded version of Figure 2.**





\begin{table}
\begin{tabular}{l l p{284.5pt}} \hline \hline  & **Reference** & During the 1980s he worked on shows such as Taxi, Cheers, and The Tracy Ullman Show. \\ (1) & Gemini 1.5 Pro & I am learning sign language. \\  & Ours (zero-shot) & In the 1980s, she worked in theaters like taxesi, cheesy, and tracy. \\  & Ours (signer-indep ft) & During the 1980s, he worked in theaters such as Axios, Chruas, and the Tracy Ultra Show. \\  & Ours (signer-dep ft) & During the 1980s, he worked in theaters like taxesi, Cheruns, and the Tracy Ullman Show. \\  & Human & During the 1980s, they worked in shows like Taxi, Cheers, and The Tracy Ullman show. \\ \hline  & **Reference** & The rise of new technologies allows us to see and investigate brain structures and processes never seen before. \\ (2) & Gemini 1.5 Pro & You must think! Use your brain! \\  & Ours (zero-shot) & There is a new technique to detect brains and vision. \\  & Ours (signer-indep ft) & The increase in new technologies allows a better understanding of the brain and its visual properties. \\  & Ours (signer-dep ft) & The increase in new technology allows the researchers to recognize brain athletics and brain memory. \\  & Human & The rise of new technology allows for investigation of brain structure and processes never seen before. \\ \hline  & **Reference** & The Articles required unanimous consent from all the states before they could be amended and states took the central government so lightly that their representatives were often absent. \\  & Gemini 1.5 Pro & It is September. It’s time to go back to school! \\  & Ours (zero-shot) & The law requires all states to agree on a standard and that it is a legal requirement. \\  & Ours (signer-indep ft) & The law requires that all states must agree on the same legal regulations. \\  & Ours (signer-dep ft) & The law requires all states to agree on the same regulations that are not legal. \\  & Human & The article mandated all states to create a uniform agreement as an amendment to the law for revisions. The states look at the process as minor and many of the state representatives did not make an effort to attend those meetings. \\ \hline \hline \end{tabular}
\end{table}
Table 6: **Qualitative examples from our sentence-level baselines for ASL to English translation on FLEURASL. Examples selected randomly without cherrypicking. In (2), see that Gemini 1.5 Pro does translate “brain”, but this is because the sign is iconic (pointing to the head). See Appendix E for the complete set of examples from the human baseline.**

* [145] M. C. B. Green, A. J. H. Sears, and R. E. Stanley, "The New 



**Figure 10.** The number of the total number of events in a single run is shown as a function of time, and the number of events in a single run is shown as a function of time.

**Figure 21.** The number of events in a single run is shown as a

* [1] The game started at 10am and the weather was perfect, even though in the morning it was a bit hazy, but has since then disappeared. The rugby game was a 7-on-7 and they followed a tournament bracket. South Africa progressed nicely in the tournament and won 26-0 against Zambia, who was 5th.
* [2] The game started at 10am and the weather was perfect, even though in the morning it was a bit hazy, but has since then disappeared. The rugby game was a 7-on-7 and they followed a tournament bracket. South Africa progressed nicely in the tournament and won 26-0 against Zambia, who was 5th.
* [3] In the beginning of the game, it was a bit rough but eventually smoothed over. South Africa, regardless, improved throughout the tournament and became stronger as they progressed. Their defense worked seamlessly together and handled the ball nicely. The team showed solidarity. That's the team that beat the other. Some of them ventured off and joined the animals while others were isolated. There are discussions on the importance of socializing. It always goes back to children who grow up with no exposure to healthy interactions with adults, and this can happen for various reasons, some being neglect, bad luck, or just abuse. When children don't get social exposure, they become what you call feral, or "wild child". Isolating their children may be the result of some people being ashamed of these children because of their mental state or their physical appearance. Sometimes it's their own parents doing this. What happens is they many sometimes escape captive because of trauma from mental abuse, sometimes severe. Sometimes these children escape and end up living with animals, being the only human. There is a trend that these children will mimic the body language and behavior of the animal they've interacted with. Sometimes these feral children are afraid of other humans while others do not care to make any connections with humans. The other team, the same as Fernando Alonso, drove and eventually spun out. He had to pull over to the pit stop because one of his tires was bent inward. They had to remove the car from the race. In the car race, Giancaultdo Fischla spun out and his car was damaged. He was out. The other team, like him, Fernando Alonso, also spun out of control and he had to pull over to the pit stop. One of the tires was bent inward and they determined that the driver was also out of the race. Another car, Michael Schumacher, after the other two were already out, was also out because his suspensions were not functioning properly. Today many Sami work in their world as their job. Tourism, for example, Sappmi is a cash cow as a form of unification. Caring and nurturing reindexer to ensure growth is important. Sami people and their culture depend on reindeer. It just so happens that other work fields are also associated with reindeer. It is not intentional for all Sami people to be involved with reindeer but living from fishing and hunting included the reliance and use of reindeer. Now in today's world, many Sami workers in tourism, for example, bring in a lot of money, Sappmi, and they are unified by that too.

 



**human prediction**

The priest John Howard said the act is for protecting the hospital from worsening. They cherish it and want to put up 45 million AUD. The minister Stephen Harper agreed he will send his government's Clean Air Act to all political parties to be looked over. There was a brief 25-minute meeting last week Tuesday with NDP's leader, Jaclyn Pmo, and Jacklyn said the act needed to be reviewed and revised because it doesn't suit their conservative environment. Understand that the government has been involved with funding for Mersey Hospital in Devevomport, Tasmania. The government MP oversaw the whole process and came to the conclusion that the NDP intentionally shifted their focus due to elections coming up in November. Something was not right. However, the priest John Howard clarified that the act is for protecting the hospital from worsening. They cherish it and want to send forth 45 million AUD.

\begin{tabular}{p{170.9pt} p{368.3pt}} \hline \hline
1 & If you booked your flights and accommodation for 2020 before the postponement was announced, you may have a tricky situation. \\ \hline  & The Paralymplics will take place from 24 August to 5 September 2021. Some events will be held in other locations throughout Japan. Tokyo will be the only Asian city to have hosted two summer Olympics, having hosted the games in 1964. If you booked your flights and accommodation for 2020 before the postonment was announced, you may have a tricky situation. Cancellation policies vary, but as of late March most coronavirus-based cancellation policies don’t extend to July 2020, when the Olympics had been scheduled. It's expected that most event tickets will cost between Z2,500 and Y130,000, with typical tickets costing around Y7,000. \\ \hline
1 & The rise of new technologies allows us to see and investigate brain structures and processes never seen before. \\ \hline  & Neurobiological data provide physical evidence for a theoretical approach to the investigation of cognition. Therefore it narrows the research area and makes it much more exact. The correlation between brain pathology and behaviour supports scientists in their research. It has been known for a long time that different types of brain damage, traumas, lesions, and tumours affect behaviour and cause changes in some mental functions. The rise of new technologies allows us to see and investigate brain structures and processes never seen before. This provides us with a lot of information and material to build simulation models which help us to understand processes in our mind. \\ \hline
1 & Courier companies are well paid for delivering things quickly. Frequently, time is very important with business quickly. Frequently, time is very important with business on some routes, the larger companies have their own planes, but for other routes and smaller firms there was a problem. If they sent things by air freight, on some routes it may taken days to get through unloading and customs. The only way to get it through faster was to send it as checked luggage. Airline regulations will not allow them to send luggage without a passenger, which is where you come in. \\ \hline \hline \end{tabular}

The minister Stephen Harper agreed he will send his government's Clean Air Act to all political parties to be looked over. There was a brief 25-minute meeting last week Tuesday with NDP's leader, Jaclyn Pmo, and Jacklyn said the act needed to be reviewed and revised because it doesn't suit their conservative environment. Understand that the government has been involved with funding for Mersey Hospital in Devevomport, Tasmania. The government MP oversaw the whole process and came to the conclusion that the NDP intentionally shifted their focus due to elections coming up in November. Something was not right. However, the priest John Howard clarified that the act is for protecting the hospital from worsening. They cherish it and want to send forth 45 million AUD.

Have you already booked a flight and hotel for a trip in 2020? Before they had announced postponement, maybe you were faced with challenges and how to resolve them.

The Paralympics will be hosted from August 24th to September 5th, 2021 in Japan. Tokyo is the only Asian city that will have hosted two summer games, the first time being in 1964. If you already booked a flight and hotel for 2020 before the announcement of the postponement, maybe you were faced with challenges and what to do. The calculation policies were all over the place. Recently in March, they had determined that during the time of COVID affecting cancellations, it doesn't apply to or affect their schedule announced in July 2020. It is expected that most tickets will cost 2,500 to 130,000 yen, with the most average cost being 7,000 yen.

The rise of new technology allows for investigation of brain structure and processes never seen before.

Neurobiological data provides evidence of theoretical approaches and investigation related to cognition, more specifically for the research area to be exact. There has been an established connection between brain pathology and behavior that has supported science and their research so far. It has already been proven that brain damage, trauma, lesions and tumors affect behaviors. These affect some changes to occur in mental functions. The rises in new technology allows for investigation of brain structure and processes never seen before. It provides new information and resources to build a model that we can mimic to help us understand the brain processes that are happening.

Courier companies are paid well for fast delivery. What happens often is these things are time sensitive such as documents related to business, materials and extra parts for quick repair.

Courier companies are paid well for fast delivery. Oftentimes, these things being delivered are time sensitive such as business documents, materials and extra materials for quick repair. Larger companies have the luxury of convenient route options such as flights while others have limited rooms, like smaller companies with less options. Problems may arise for them. When these companies ship via flight, some will have to go through customs and investigations prolonging the process, so they expedite the process by putting these items in their own luggages onto a flight. However, flight regulations do require that luggages travel with their person and that is what the role of a courier is



**Figure 12.** The largest tournament of the year takes place in December at the polo fields in Las Canitas.

\begin{tabular}{c p{340.7pt}} \hline \hline
2 & The largest tournament of the year takes place in December at the polo fields in Las Canitas. \\ \hline  & Argentina is well known for having one of the best polo teams and players in the world. The largest tournament of the year takes place in December at the polo fields in Las Canitas. Smaller tournaments and matches can also be seen here at other times of the year. For news on tournaments and where to buy tickets for polo matches, check Asociacion Argentina de Polo. \\ \hline   2 & After seeing the horrors and advcitics of war during World War I, nations desired to avoid such a situation again in the future. \\ \hline  & During the 1920s, the prevailing attitudes of most citizens and nations was that of pacifism and isolation. After seeing the horrors and advcitics of war during World War I, nations desired to avoid such a situation again in the future. \\ \hline   2 & Children are placed in Foster Care for a wide variety of reasons that range from neglect, to abuse, and even to extortion. \\ \hline  & Children are placed in Foster Care for a wide variety of reasons that range from neglect, to abuse, and even to extortion. \\ \hline  & Children are in foster care for different reasons like neglect, sons that range from neglect, to abuse, and even for forcing, threat- No child should ever have to grow up in an environment that is not nurturing, caring, and educational, but they do. We perceive the Foster Care System to be a safety zone for these children. Our foster care system is supposed to provide safe homes, loving caregivers, stable education, and reliable health care. Foster care is supposed to provide all the necessities that were lacking in the home they were previously taken from. \\ \end{tabular}




**Figure 10.**

* 2: One of the most noteworthy recent examples of this was the North Atlantic campaign of WWII. The Americans were trying to move men and materials across the Atlantic Ocean to help Britain. Using ships to transport goods is by far the most efficient way to move large amounts of people and goods across oceans. The job of navies has traditionally been to ensure that your country maintains the ability to move your people and goods, while at the same time, interfering with your enemy's ability to move his people and goods. One of the most noteworthy recent examples of this was the North Atlantic campaign of WWII. The Americans were trying to move men and materials across the Atlantic Ocean to help Britain. At the same time, the German navy, using mainly U-boats, wasting to stop this traffic. Had the Allies failed, Germany probably would have been able to conquer Britain as it had the rest of Europe. This term derives from ancient familiarity with Bed-bugs, which are insects highly adapted to parasitize humans. The word bug comes from the word "bed bug", which is an insect that pierces and burrows under the skin, and becomes a parasite. People who study insects were looking over a subgroup of insects that included the word 'bug'. The word comes from an insect called 'Bed Bugs' that would pierce and burrow under the skin, and become a parasite. There are two different types of these bugs, Assassin Bugs and Bed Bugs, which tend to adapt and remain in the body. They essentially nest in the body for as long as they can. Thoughs of years in Europe and in their different regions, Christians stayed together regardless of different languages or customs. In Europe's historical time period where catholic churches were wealthy and powerful, all eyes were on them. In the thousands of years in Europe and their different regions, Christians stuck together regardless of their different languages or customs. Religion was powerful and had influence everywhere such as monarchy and even peasants. Christians had a belief system and one part says that the wealthy shall help the poor and reduce their suffering. With this in mind, the Church would save their money with the intention of reducing suffering in the poor. Over the thousands of years, the central authority figure in churches everywhere was in Rome and they focused on the power of money and how it had influence on others. It caused people to question whether they were truly following their belief system. Plants create oxygen, of which people use and breathe in. Then people breathe out carbon dioxide which plants need. It is a transactional process. Plants create oxygen of which people breathe in. Then people breathe out, they are letting out carbon dioxide which plants need. Plunts create oxygen of which people breathe in. Then when people breathe out, they are letting out carbon dioxide which plants need. It's a transactional process. Plants rely on the sun where it helps them produce their own food, and this process is called photosynthesis. Plants also provide shades, housing, materials, clothes, and even most food come from plants. Animals cannot survive without plants.



* **Reference**
* **3**: They begin as funnels descending from storm clouds, and become "tornadoes" when they touch the ground.
* **4**: A tornado is a spinning column of very low-pressure air, which sucks the surrounding air inward and upward. They generate high winds (often 100-200 miles/hour) and can lift heavy objects into the air, carrying them as the tornado moves. They begin as funnels descending from storm clouds, and become "tornadoes" when they touch the ground.
* **5**: The setting might be an historic old building with antique furnishings, manicured grounds and a swimming pool.
* **6**: In developed countries today, providing deluxe bed and breakfasts has been raised to a sort of art-form. At the top end, B&Bs obviously compete mainly on two main things: bedding and breakfast. Accordingly, at the finest such establishments one is apt to find the most luxurious bedding maybe a handmade quilt or antique bed. Breakfast may include seasonal delights of the region or the host's speciality dish. The setting might be an historic old building with antique furnishings, manicured grounds and a swimming pool.
* **7**: Chiao and Sharipov reported being a safe distance from the attitude adjustment thrusters.
* **8**: The station maintained its attitude, despite the loss of a gyroscope earlier in the space station mission, until the end of the spacecraft. Chiao and Sharipov reported being a safe distance from the attitude adjustment thrusters. Russian ground control activated the jets and normal attitude of the station was regained.
* **9**: Bird flu, or more formally avian influenza, can infect both birds and mammals.
* **10**: Bird flu, or more formally avian influenza, can infect both birds and mammals. Fewer than a thousand cases have ever been reported in humans, but some of them have been fatal. Most have involved people who work with poultry, but there is also some risk to birdwatchers.
* **11**: On the accordion, to get extra volume, you use the bellows with more pressure or speed.
* also try not to make much extraneous motion with your fingers. This way, you will tire yourself out as little as possible. Remember there’s no need to hit the keys with a lot of force for extra volume like on the piano. On the accordion, to get extra volume, you use the bellows with more pressure or speed.
* **13**: While immigration check is usually absent or a formality when you arrive in your homeland, customs control can be a hassle.
* **14**: While immigration check is usually absent or a formality when you arrive in your homeland, customs control can be a hassle. Make sure you know what you can and cannot bring in and declare anything over the legal limits.
* **15**: When you arrive in your homeland, immigration customs will inspect you. Sometimes they are casual while at other times they are rigid and follow protocol. Custons regulations can be a nuisance.
* **16**: When you arrive in your homeland, immigration will do checks where it is sometimes casual but at other times more of a formality. Customs regulations can be a nuisance. Just make sure you know what you can and can't bring and announce anything that is above the legal limit.

 

* [150] M. Crane, R. G. Leigh, and J. Silk, "A new approach to the detection of a large number of events in the search for a single event from a single event using a single event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event event with an event with an event with an event with an event with an event event with an event with an event with an event event with an event with an event with an event event with an event with an event event with an event with an event with an event event with an event with an event event with an event with an event event with an event with an event event with an event with an event event with an event with an event event with an event with an event event with an event with an event event with an event with an event event with an event with an event event with an event with an event event with an event with an event event with an event with an event event with an event with an event event with an event with an event event with an event with an event event with an event with an event event with an event with an event event with an event with an event event with an event with an event event with an event with an event event with an event with an event event with an event with an event event with an event with an event event with an event with an event event with an event with an event event with an event with an event event with an event with an event event with an event with an event event with an event with an event event with an event with an event with an event event with an event with an event with an event event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event with an event an event with an event with an event with an event with an event an event with an event with an event with an event an event with an event with an event an event with an event with an event an event with an event with an event an event with an event with an event an event with an event with an event an event with an event with an event an event with an event with an event an event with an event with an event an event with an event with an event an event with an event with an event an event with an event with an event an event with an event with an event an event with an event with an event an event with an event with an event an event with an event with an event an event with an event with an event an event with an event with an event an event with an event with an event an event with an event with an event an event with an event with an event an event with an event with an event an event with an event with an event an event with an event with an event an event with an event with an event an event with an event with an event an event with an event with an event an event with an event with an event an event with an event with an event an event with an event with an event an event with an event with an event an event with an event with an event an event with an event with an event an event with an event with an event an event with an event with an event an event with an event with an event an event with an event with an event an event with an event with an event an event with an event with an event an event with an event with an event an event with an event with an event an event with an event with an event an event with an event with an event an event with an event with an event an event with an event with an event an event with an event with an event an event with an event with an event an event with an event with an event an event with an event with an event an event with an event with an event with an event an event with an event with an event with an event an event with an event with an event with an event with an event an event with an event with an event with an event with an event 

* [15] M. C. Davis, A. D. K. Ellis, and R. G. Wilson, "The evolution of the universe: The history of the universe," _Astrophys. J. Suppl._**463**, 207-218, doi:10.1093/s10052-007-0193-z
* [16] S. F. King, "A review on the structure of the universe," in _Proceedings of the Royal Society of London_, vol. 10, no. 1, pp. 103-118, doi:10.1142/S02147 

* [150] M. C. Davis, J. A. Harvey, and R. H. Sellwood, "A survey of the northern sky: a survey of southern-western sky in the north Galactic center," _Journal of Astronomical Society B: Volume I_, vol. 347, no. 2, pp. 869-874, 1996.

[MISSING_PAGE_POST]