dataesr/scientific-paragraphs-categorization huggingface dataset:
Our main goal is to study how context impacts moment localization with temporal language, so we use the LSTM features used on the original DiDeMo dataset. Similarity f S. Given video f V and language f L features, we consider three ways to encode similarity between the features. Like Hendricks et al. (  2017 ), we consider a distance-based similarity f S = |f V -f L | 2. Second, we consider a fused-feature similarity (mult) where the Hadamard product f V f L between the two features are passed to a MLP. We also explore unit normalizing features before the Hadamard product (normalized mult). Finally, we consider the similarity (TALL similarity) which consists of the concatenation (f V , f L , f V f L , f V + f L ) and then passed to a MLP. Context moments T τ. We consider three sets of context moments.
We also consider the alignment loss used in TALL (TALL loss) which is the sum of two log-logistic functions over positive and negative training query/moment pairs (intra-video negatives are used). Supervising context moments. For the temporal sentences in our newly collected dataset (Section 4), we have access to the ground-truth context moment during training. Thus, we can contrast a weakly supervised setting in which we optimize over the unknown latent context moments during learning and inference to a strongly supervised setting. Implementation details. Candidate base and context moments coincide to the pre-segmented fivesecond segments used when annotating DiDeMo. Moments may consist of any contiguous set of five-second segments. For a 30-second video partitioned into six five-second segments, there are 21 possible moments.
The TEMPO Dataset We collect the TEMPOral reasoning in video and language (TEMPO) dataset based off the recently released DiDeMo dataset. Our dataset consists of two parts: TEMPO -Template Language (TL) and TEMPO -Human Language (HL). We create TEMPO -TL using language templates to augment the original sentences in DiDeMo with tem- * https://people. eecs. berkeley. edu/ ˜lisa_anne/tempo. html poral words. The template allows us to generate a large number of sentences with known ground truth base and context moments. However, template language lacks the complexity of human language, so we then collect an additional fully userconstructed dataset, TEMPO -HL, consisting of sentences that contain specific temporal words. Temporal Words in Current Datasets. We first analyze temporal words which occur in current natural language moment retrieval datasets.
We consider temporal adjectives, adverbs, and prepositions found both by closely analyzing moment-localization datasets and consulting lists containing words which belong to different parts of speech. In particular, we rely on the preposition project (Litkowski and Hargraves, 2005) † to scrape relevant temporal words. Table  2  shows example temporal words and the number of times they occur in each dataset (TACoS  (Regneri et al. , 2013) , Charades  (Gao et al. , 2017 ), DiDeMo (Hendricks et al. , 2017) ). Though all moment localization datasets use temporal words, they do not contain enough examples to reliably train and evaluate current models. Additionally, we observe that temporal words which are frequently used when describing video segments are different than those commonly used in text without video grounding.
For example, in  Pratt-Hartmann (2004) , "during" is a common example, but we observe that "during" is infrequently used when describing video. Of temporal words, we focus on the four most common words, "before", "after", "then", and "while" when creating our dataset. TEMPO -Template Language. To construct sentences in TEMPO-TL, we find adjacent moments in the DiDeMo dataset and fill in template sentences for "before", "after", and "then" temporal words. For "before", we use two templates: "X before Y " and "Before Y , X", where X and Y are sentences from the original DiDeMo dataset. Likewise for "after", we consider the templates "X after Y " and "After Y , X". For "then" we only consider one template, "X then Y. " TEMPO -Human Language. Though the template dataset is an interesting testbed for understanding temporal language, it is difficult to replicate the interesting complexities in human language.
For example, when writing long sen-  The adult hands the little boy a stick. The adult hands the little boy a stick then they begin to walk. The boy and adult stop before adult bends over and hands child a short stick. The girl looks at the camera and waves The little girl turns and waves at the camera while on her skates. After the girl waves at the camera she continues to skate. tences with temporal prepositions, humans frequently make use of language structure such as coreference to form more cohesive statements. To collect annotations, we follow the protocol in Hendricks et al. (  2017 ) and segment videos into 5-second temporal segments. After collecting descriptions, we ensure descriptions are localizable by asking other workers to localize each moment. To collect data for "before", "after", and "then", we ask annotators to describe a segment in relation to a "reference" moment from the DiDeMo dataset.
For example, if the DiDeMo dataset includes a localized phrase like "the cat jumps", annotators write a sentence which refers to the segment "the cat jumps" using a specific temporal word. We provide both the phrase ("the cat jumps") and the reference moment to annotators, and the annotators provide a sentence describing a new moment which references the reference moment. TEMPO-HL includes unique properties which are hard to replicate with template data. Figure  3  depicts the base moment provided to workers, as well as descriptions from TEMPO-HL. In Figure  3 , the description "The adult hands the little boy the stick then they walk away" includes an example of visual coreference ("they"). We note that use of pronouns is much more prevalent in TEMPO-HL, with 28. 1% of sentences in TEMPO-HL including pronouns ("he", "she", "it") in contrast to 10.
Experiments Evaluation Method. We follow the evaluation protocol defined for the DiDeMo dataset (Hendricks et al. , 2017) over all possible combinations of the five-second video segments. We report rank at one (R@1), rank at five (R@5), and mean intersection over union (mIOU) using their aggregator over three out of the four human annotators. We compare our models on TEMPO-TL, TEMPO-HL, and the DiDeMo dataset. When training our models, we combine the DiDeMo dataset with TEMPO-TL or TEMPO-HL. This enables our model to concurrently learn to localize the simpler DiDeMo sentences with more complex TEMPO sentences. Baselines. We compare to the two recently proposed approaches for video moment localization:  MCN (Hendricks et al. , 2017)  and TALL  (Gao et al. , 2017). We adapt the implementation of TALL  (Gao et al.
, 2017)  to the DiDeMo dataset in three ways. First, we do not include the temporal localization loss required to regress to specific start and end points as DiDeMo, and thus also TEMPO, is pre-segmented, so the model does not need to compute exact start and end points. Second, the original TALL model uses C3D features. For a fair comparison we train both models with the same RGB and flow features extracted as was done for the original MCN model. Finally, the MCN model proposes temporal endpoint features (TEF) to indicate when a proposed moment occurs within a video. We train TALL with and without the TEF and show that TEF improves performance on the original DiDeMo dataset. Ablations. To ablate our proposed latent context, we compare to other models which share the same MLLC base network.
We consider the MLLC model with global context and before/after context. We also train a model with weakly supervised (WS) latent context and strongly supervised (SS) latent context. We also train models both with and without context TEF (conTEF). The MLLC Base Model. We first ablate our MLLC base model (Table  3 ). We train our models on TEMPO-TL and DiDeMo and evaluate on the original DiDeMo dataset. All models are trained with global context. We find that the ranking loss is preferable on the DiDeMo dataset (compare lines 1 and 2) and that TALL-similarity performs better than the distance based similarity of the MCN model (compare lines 1 and 5). A simpler version of the TALL-similarity, in which the concatenated element wise multiplication, element wise sum, and concatenation is replaced by a single normalized elementwise multiplication, increases R@1 by almost one point and increases mIoU by over two points (compare lines 5-7).
If a model has good temporal reasoning, then if it can localize a reference moment "the dog jumps" it should be easier for the model to localize the moment "the dog sits after the dog jumps". To test whether models are capable of this, we look at only sentences in TEMPO -TL where the model has correctly localized the cor-TEMPO -Template Language (TL) DiDeMo Before After Then Average R@1 mIoU R@1 mIoU R@1 mIoU R@1 mIoU R@1 R@5 mIoU Table  5 : Difference between performance on full dataset and set on which reference moments are localized properly for different methods on TEMPO-TL. responding context moment in DiDeMo (Table  5 ). We report the difference in performance when considering only sentences in which temporal context was properly localized and all sentences. On our model, performance on all three temporal word types increases when the context moment can be properly localized.
In particular, our model has the strongest performance for all sentence types considering the mIoU metric. In addition to performing better on temporal words, our model also performs better on the original DiDeMo dataset. As was seen in TEMPO-TL, including before/after context performs better than our model trained with global context for both "before" and "after" words. The final row of Table  6  shows an upper bound in which the ground truth context is used at test time instead of the latent context. We note that results improve for "before", "after", and "then", suggesting that learning to better localize context will improve results for these sentence types. Localizing Context Fragments. TEMPO-HL sentences can be broken into two parts: a basesentence fragment (which refers to the base moment), and a context-sentence fragment (which refers to the context moment).
Likewise, for the moment "After zooming in to the dog, the dog darts across the grass and into the woods", the dog darts towards the woods twice (at the beginning of the video and at the end). Our model properly localizes the moment when the dog runs towards the forest the second time as well as the context fragment "zooming in on dog" when localizing the moment. Discussion. We show promising results on both TEMPO-TL and TEMPO-HL, but there is potential improvement for building better frameworks for understanding temporal language. In Table  6 , strongly supervising context at test time improves overall results, suggesting that models which can better localize context text will outperform our current model. Though TEMPO and DiDeMo have over 60,000 sentences combined, visual content is quite diverse.
Integrating outside data sources (e. g. , image retrieval and captioning) could possibly improve results on moment localization, both with and without temporal language queries. Additionally, in Table  5 , even when the MLLC model can properly localize context, it does not always properly localize temporal sentences indicating that improved temporal reasoning can also improve our results. We believe our dataset, analysis, and method are an important step towards better moment retrieval models that effectively reason about temporal language. A TEMPO-Template Language Figure  5  shows examples from our TEMPO-TL dataset. At the top, we show two sentences from the original DiDeMo dataset ("the cross is seen for the first time" and "window is first seen in room"). A "then" moment (left, green) is created by concatenating the two adjacent moments from the DiDeMo dataset and combining the sentences using the word "then" ("The cross is seen for the first time then window is first seen in room.
B Qualitative Examples on TEMPO-Human Language In this section we show qualitative results for when multiple sentences for the same video are localized properly (Figure  6 ), examples in which our model properly localizes sentence queries , and failure cases (Figure  12 ). Comparing Different Temporal Words for the Same Video. Figure  6  shows an example where the original DiDeMo moment is localized correctly ("a cat jumps up and spazzes out") as well as temporal sentences "the cat sniffs the floor before it jumps ip [sic] and spazzes out", "a cat jumps up and spazzes out then it goes under the counter", and "the cat put it's head under the shelf after it jumps up and spazzes out. " Our MLLC model is able to properly localize each sentence type. Notice that when the annotators construct temporal sentences, they include pronouns like "it" as opposed to repeating the word "cat" multiple times.
This observation may partially explain why the MLLC model does better on the original DiDeMo dataset. Finally, Figure  12  shows interesting failure cases. Figure  12  (top) shows an example where the context was localized properly, but the moment was not correctly localized. This could be in part because the query is particularly complex and uses two temporal words ("then" and "before"). Finally, Figure  12  (bottom) shows a failure case for the word "after" where the context moment is temporally related to the localized moment in a sensible way; for an "after" sentence we expect the context to occur before the localized moment. The cross is seen for the first time. Window is first seen in room. The cross is seen for the first time before window is first seen in room. The cross is seen for the first time then window is first seen in room.
Several methods have been explored to reduce or compensate for end-toend latency, either controlling the entire graphical rendering pipeline  [8] , building homemade hardware devices and handling systems  [10]  or using software compensation by trying to predict the future positions based on previous positions, velocities and accelerations obtained by derivatives  [6]. We present TurboMouse  [1] , a hybrid hardware and software compensation of end-to-end latency on desktop computers when operated using a computer-mouse, and we contribute a prediction technique specifically designed for indirect pointing. Our technique is based on the prediction of pointer position based on the velocity measured by the mouse and the acceleration from an accelerometer embedded in the mouse. Our assumption is that the hardware acceleration measured by the accelerometer is more precise and less noisy than a computed one from the velocity, which is captured discretely by the optical sensor of the mouse controller.
Software The mouse controller and Arduino board send their data to a host computer via USB that runs a dedicated software (written in C++ with the Qt Framework) to handle these signals and predict the next position of the mouse cursor. Merging input data: Acceleration inputs are transmitted to the computer every 1ms using HIDAPI in a dedicated blocking-thread. Each event received is timestamped and buffered by our software. The Arduino also records and sends a timestamp to the host-computer. Mouse inputs are provided at 1 kHz by the mouse controller and captured by the libpointing library which allows us to get raw informations (dx [counts], dy [counts], timestamp [ns], buttons) at low level. We then associate each mouse input with the most recent acceleration values stored in the buffer. Supporting transfer functions: Desktop systems commonly use non-linear transfer functions that dynamically adjust a control-display gain (G) that will impact cursor displacement  [4]  depending on the input velocity.
Thymic carcinomas Thymic carcinomas (TC) are a less common subset of TETs that entail a more aggressive clinical course. These tumors histologically resemble common epithelial tumors. To better understand this subset, we performed a focused analysis on the TC samples within our dataset (n=10). Mutational analysis did not identify any recurrently mutated genes in this subset (Figure  6A ). When looking at arm-level sCNAs however, we did observe 8 of 10 samples having a loss of Chromosome 16q. This observation has been previously reported in TC  (Zettl et al. , 2000). Evaluation of the Sanger Gene Census revealed several tumor suppressors in this region, including: CYLD, CBFB, CDH1, CDH11, CTCF, ZFHX3. When comparing the tumor mutation burden (TMB) between TC vs all other thymoma histologies, we observed a significant increase in TMB in TC samples (p=5.
A recent publication by Lee et al. using the publically available TCGA TET dataset also demonstrated separation of TETs into four clusters defined by: GTF2I mutations, T-cell signaling, chromosomal stability, and chromosomal instability  (Lee et al. , 2017). GTF2I L424H mutations are unique to TETs and is the most common mutation in this tumor type. This mutation was observed in 100% of the type A and 70% of type AB thymomas. Mutations in GTF2I have been described rarely in other tumor types and are present at different codons. RNA-seq identified higher expression of genes involved in cell morphogenesis, receptor tyrosine kinases, retinoic acid receptors, neuronal processes, as well as the WNT and SHH signaling pathways in the GTF2I mutant tumors. These results are similar to those observed when Gtf2ird1 was knocked out in a mouse model  (Corley et al.
, 2016). GTF2IRD1 is a GTF2I family member, located near GTF2I, which was also upregulated in our dataset in GTF2I mutants. From this study, we also observe clonal, recurrent drivers in HRAS and NRAS. These potent oncogenes may play a role in determining prognosis, to be assessed in future studies with longitudinal follow-up. Also of note, we do not observe any viral etiology as reported before. Autoimmunity is a hallmark of thymomas, with TAMG occurring in close to 30% of cases  (Zekeridou et al. , 2016). Thymoma represents the only TCGA tumor type to be analyzed that has a strong association with an autoimmune disease. This analysis demonstrates a significant association of aneuploidy with a cancer-associated autoimmune disease. However, still unknown is why TAMG patients have a limited spectrum of autoantibodies to the AChR, titin, and ryanodine receptors (RYR1, RYR2)  (Gilhus et al.
• A TCGA-specific letter that the IRB does not consider the use of these data and materials to be human subjects research. The was most common for collections in which the donors were deceased. Cases were classified by the submitting institution in accordance with the World Health Organization (WHO 2004) categories of type A, AB, B1, B2, B3, or TC. Pathology quality control was performed on each tumor specimen from a frozen section slide prepared by the BCR. Hematoxylin and eosin (H&E) stained sections from each sample were subjected to independent pathology review to confirm that the tumor specimen was histologically consistent with the reported thymic epithelial tumor type. The percent tumor nuclei, percent necrosis, and other pathology annotations were also assessed. Tumor samples with ≥60% tumor nuclei (with exception for WHO Type B1 tumors which have high lymphocytic infiltration obviating the ability to have ≥60% tumor nuclei) and ≤20% necrosis were submitted for nucleic acid extraction.
, 2012). As a source of leukocyte DNA methylation level, we used DNA methylation data of peripheral blood mononuclear cells (PBMC) from six healthy donors  (Reinius et al. , 2012 ) (GSE35069). Additional Analyses-Fisher's exact test was used to test for associations of DNA methylation clusters with mRNA expression clusters and significantly mutated genes. Analyses described above as well as plots including heat maps and scatterplots were carried out in R using standard methods and customized routines. miRNA microRNA Libraries and Sequencing-We generated microRNA sequence (miRNAseq) data for 117 tumor samples using methods described previously except that 1ug of total RNA (at 250ng/uL) was used as input instead of messenger RNA-depleted RNA  (Chu et al. , 2016). Briefly, reads were aligned to the GRCh37/hg19 reference human genome, and read count abundance was annotated with miRBase v16 stemloops and mature strands.
While the read counts included only exact-match read alignments,. bam files at CGHub (cghub. ucsc. edu) (Wilks et al. , 2014)  include all sequence reads. We used miRBase v20 to assign 5p and 3p mature strand (miR) names to MIMAT accession IDs. Unsupervised Clustering of miRNA Mature Strands-To identify subtypes within the THYM cohort we used hierarchical clustering with pheatmap v1. 0. 2 in R. The input was a reads-per-million (RPM) data matrix for the 303 (top 25%) miRBase v16 5p or 3p mature strands that had the largest variances across the cohort. We transformed each row of the matrix by log10(RPM + 1), then used pheatmap to scale the rows. We used Ward. D2 for the clustering method with Pearson correlation and Euclidean as the distance measures for clustering the columns and rows respectively.
RNAseq RNA Library Construction, Sequencing, and Analysis-One μg of total RNA was converted to mRNA libraries using the lllumina mRNA TruSeq kit  (RS-122-2001 or RS-122-2002)  following the manufacturer's directions. Libraries were sequenced 48×7×48bp on the Illumina HiSeq 2000 as previously described  (Cancer Genome Atlas Research, 2012). FASTQ files were generated by CASAVA. RNA reads were aligned to the hg19 genome assembly using MapSplice 0. 7. 4  (Wang et al. , 2010b). Gene expression was quantified for the transcript models corresponding to the TCGA GAF2. 1 (http://tcgadata. nci. nih. gov/docs/GAF/GAF. hg19. June2011. bundle/outputs/TCGA. hg19. June2011. gaf), using RSEM  (Li and Dewey, 2011)  and normalized within-sample to a fixed upper quartile. For further details on this processing, refer to Description file at the DCC data portal under the V2_MapSpliceRSEM workflow (https://tcga-data.
nci. nih. gov/tcgafiles/ftp_auth/ distro_ftpusers/anonymous/tumor/thym/cgcc/unc. edu/illumina hiseq_rnaseqv2/rnaseqv2/ unc. edu_THYM. IlluminaHiSeq_RNASeqV2. mage-tab. 1. 0. 0/DESCRIPTION. txt) or our alignment pipeline summary at CGHUB (https://cghub. ucsc. edu/docs/tcga/ UNC_mRNAseq_summary. pdf). Quantification of genes, transcripts, exons and junctions can be found at the TCGA Data Portal (https://tcga-data. nci. nih. gov/tcga/). Unsupervised Clustering-A set of genes that were both highly expressed and variably expressed was identified and used for clustering. After restricting to genes with at least 75% non-zero RSEM values, the genes with the 1000 highest median absolute deviation (MAD) values were chosen. RSEM values identically equal to zero were replaced into smallest nonzero value.
, 2001)  was used to identify differentially expressed genes between different expression subtypes and groups of patients defined by clinical characteristics using 1000 permutations. We then used the DAVID annotation database  (Huang et al. , 2007a; Huang et al. , 2007b)  to identify enriched pathways. Supervised Clustering-The ClaNC R package  (Dabney, 2006)  was used to identify genes whose expression patterns characterize the RNA subtypes. Gene Fusion Detection-In addition to quantifying gene expression, RNA sequencing can detect structural variants, including alternate splicing, intra-chromosomal fusions, and inter-chromosomal fusions. Two algorithms were used to identify gene fusions: AccuFusion (In-house tool), MapSplice  (Wang et al. , 2010b). Virus Detection-In addition to quantifying gene expression, RNA sequencing can detect viral transcripts using virus database including HPyV 6, 7, 9, 10, 12 and MCPyV.