Miracle-dz/UniFAR-FTUs huggingface dataset:
Rethinking Alignment in Video Super-Resolution Transformers

The alignment of adjacent frames is considered an essential operation in video super-resolution (VSR). Advanced VSR models, including the latest VSR Transformers, are generally equipped with well-designed alignment modules. However, the progress of the self-attention mechanism may violate this common sense. In this paper, we rethink the role of alignment in VSR Transformers and make several counter-intuitive observations. Our experiments show that: (i) VSR Transformers can directly utilize multi-frame information from unaligned videos, and (ii) existing alignment methods are sometimes harmful to VSR Transformers. These observations indicate that we can further improve the performance of VSR Transformers simply by removing the alignment module and adopting a larger attention window. Nevertheless, such designs will dramatically increase the computational burden, and cannot deal with large motions. Therefore, we propose a new and efficient alignment method called patch alignment, which aligns image patches instead of pixels. VSR Transformers equipped with patch alignment could demonstrate state-of-the-art performance on multiple benchmarks. Our work provides valuable insights on how multi-frame information is used in VSR and how to select alignment methods for different networks/datasets. Codes and models will be released at https://github.com/XPixelGroup/RethinkVSRAlignment.
COMPARATIVE ANALYSIS OF UNIVERSAL METHODS NO REFERENCE QUALITY ASSESSMENT OF DIGITAL IMAGES

The main purpose of this article is to conduct a comparative study of two well-known no-reference image quality assessment algorithms BRISQUE and NIQE in order to analyze the relationship between subjective and quantitative assessments of image quality. As experimental data, we used images with artificially created distortions and mean expert assessments of their quality from the public databases TID2013, CISQ and LIVE. Image quality scores were calculated using the NIQE, BRISQUE functions and their average. The correlation coefficients of Pearson, Spearman and Kendall were analyzed between expert visual assessments and quantitative scores of the image quality, as well as between the values of three compared indicators. For the experiments, the Matlab system and values of its functions niqe and brisque normalized to the range [0, 1] were used. The computation time of niqe is slightly less. The investigated functions poorly estimate the contrast of images, but the additive Gaussian noise, Gaussian blur and loss in compression by the JPEG2000 algorithm are better. The BRISQUE measure shows slightly better results when evaluating images with additive Gaussian noise, while NIQE for blurred by Gaussian. The average of the normalized values of NIQE and BRISQUE is a good compromise. The results of this work may be of interest for the practical implementations of digital image analysis.
Organisation of the mammalian genome in the nucleus

Transcription factor binding to specific DNA sequences is insufficient to explain the control of gene expression. Attention has turned to large-scale spatial organisation and functional compartmentalisation of chromosomes, and of the nucleus itself, in the quest to understand how the expression of complex genomes is regulated. Chromosomes occupy discrete domains in interphase and there has been speculation that transcription, RNA processing and RNA transport occur in a space between chromosomes which links to the nuclear pores, the inter-chromosome domain (ICD). In support of this model, studies of a small number of isolated sequences found genes to be positioned at the surface of chromosomes, independent of their transcriptional status. However, nascent transcripts and both early(gene-rich) and late-replicating (gene-poor) DNA appear to be distributed throughout chromosome territories. Consequently, the ICD compartment model has been modified to include space between the surfaces of subchromosomal domains. Conversely, it has been suggested that differential localisation of gene sequences adjacent to centromeric heterochromatin can facilitate transcriptional repression. To determine whether genes occupy specific sites within interphase chromosome territories, I have used 2D and 3D fluorescence in situ hybridisation (FISH) to investigate the spatial organisation of stretches of DNA from the short arm of human chromosome 11 (HSA1 lp), which differ in gene content. I found that gene sequences contained in 1Mb ofDNA from the moderately gene-rich WAGR (Wilms' tumour, aniridia, genitourinary, mental retardation syndrome) locus of llpl3 are not localised at the periphery of the interphase HSAllp territory as compared to intergenic, non-expressed sequences from the same region. The entire locus is positioned approximately one third of the way along the radius of the HSA1 lp domain. The organisation of this 1Mb of DNA remains unchanged in cell lines expressing different combinations of genes, indicating that transcriptional status does not play a significant role in the interphase positioning of this locus in human cell lines. In contrast, I found that DNA from gene-rich 1 lp 15.5 adopts an unusual configuration in the interphase nucleus. Sequences from this region appear to localise outside of the main body of the HSA1 lp territory. I have examined possible explanations for this extrusion of chromatin from the chromosome domain. It is not characteristic of all chromatin containing telomere
HQANN: Efficient and Robust Similarity Search for Hybrid Queries with Structured and Unstructured Constraints

The in-memory approximate nearest neighbor search (ANNS) algorithms have achieved great success for fast high-recall query processing, but are extremely inefficient when handling hybrid queries with unstructured (i.e., feature vectors) and structured (i.e., related attributes) constraints. In this paper, we present HQANN, a simple yet highly efficient hybrid query processing framework which can be easily embedded into existing proximity graph-based ANNS algorithms. We guarantee both low latency and high recall by leveraging navigation sense among attributes and fusing vector similarity search with attribute filtering. Experimental results on both public and in-house datasets demonstrate that HQANN is 10x faster than the state-of-the-art hybrid ANNS solutions to reach the same recall quality and its performance is hardly affected by the complexity of attributes. It can reach 99% recall@10 in just around 50 microseconds On GLOVE-1.2M with thousands of attribute constraints.
Dual-Branch Hybrid Learning Network for Unbiased Scene Graph Generation

The current studies of Scene Graph Generation (SGG) focus on solving the long-tailed problem for generating unbiased scene graphs. However, most de-biasing methods over-emphasize the tail predicates and underestimate head ones throughout training, thereby wrecking the representation ability of head predicate features. Furthermore, these impaired features from head predicates harm the learning of tail predicates. In fact, the inference of tail predicates heavily depends on the general patterns learned from head ones, e.g., “standing on” depends on “on”. Thus, these de-biasing SGG methods can neither achieve excellent performance on tail predicates nor satisfying behaviors on head ones. To address this issue, we propose a Dual-branch Hybrid Learning network (DHL) to take care of both head predicates and tail ones for SGG, including a Coarse-grained Learning Branch (CLB) and a Fine-grained Learning Branch (FLB). Specifically, the CLB is responsible for learning expertise and robust features of head predicates, while the FLB is expected to predict informative tail predicates. Furthermore, DHL is equipped with a Branch Curriculum Schedule (BCS) to make the two branches work well together. Experiments show that our approach achieves a new state-of-the-art performance on VG and GQA datasets and makes a trade-off between the performance of tail predicates and head ones. Moreover, extensive experiments on two downstream tasks (i.e., Image Captioning and Sentence-to-Graph Retrieval) further verify the generalization and practicability of our method. Our code is available at https://github.com/aa200647963/SGG-DHL/.
Conditional colour-magnitude distribution of central galaxies in galaxy formation models

We investigate the conditional colour-magnitude distribution (CCMD), namely the colour-magnitude distribution at fixed halo mass, of the central galaxies in semi-analytic galaxy formation model (SAM) and hydrodynamic simulations. We analyse the CCMD of central galaxies in each halo mass bin with the Gaussian mixture model and find that it can be decomposed into red and blue components nearly orthogonal to each other, a red component narrow in colour and extended in magnitude and a blue component narrow in magnitude and extended in colour. We focus on the SAM galaxies to explore the origin of the CCMD components by studying the relation between central galaxy colour and halo or galaxy properties. Central galaxy colour is correlated with halo assembly properties for low mass haloes and independent of them for high mass haloes. Galaxy properties such as central supermassive black hole mass, cold gas mass, and gas specific angular momentum can all impact central galaxy colour. These results are corroborated by an alternative machine learning analysis in which we attempt to predict central galaxy colour with halo and galaxy properties. We find that the prediction for colours of central galaxies can be significantly improved using both halo and galaxy properties as input compared to using halo properties alone. With the halo and galaxy properties considered here, we find that subtle discrepancies remain between predicted and original colour distribution for low mass haloes and that no significant determining properties are identified in massive haloes, suggesting modulations by additional stochastic processes in galaxy formation.
Bagging Regional Classification Activation Maps for Weakly Supervised Object Localization

Classification activation map (CAM), utilizing the classification structure to generate pixel-wise localization maps, is a crucial mechanism for weakly supervised object localization (WSOL). However, CAM directly uses the classifier trained on image-level features to locate objects, making it prefers to discern global discriminative factors rather than regional object cues. Thus only the discriminative locations are activated when feeding pixel-level features into this classifier. To solve this issue, this paper elaborates a plug-and-play mechanism called BagCAMs to better project a well-trained classifier for the localization task without refining or re-training the baseline structure. Our BagCAMs adopts a proposed regional localizer generation (RLG) strategy to define a set of regional localizers and then derive them from a well-trained classifier. These regional localizers can be viewed as the base learner that only discerns region-wise object factors for localization tasks, and their results can be effectively weighted by our BagCAMs to form the final localization map. Experiments indicate that adopting our proposed BagCAMs can improve the performance of baseline WSOL methods to a great extent and obtains state-of-the-art performance on three WSOL benchmarks. Code are released at https://github.com/zh460045050/BagCAMs.
Reflow: Automatically Improving Touch Interactions in Mobile Applications through Pixel-based Refinements

Touch is the primary way that users interact with smartphones. However, building mobile user interfaces where touch interactions work well for all users is a difficult problem, because users have different abilities and preferences. We propose a system, Reflow, which automatically applies small, personalized UI adaptations, called refinements -- to mobile app screens to improve touch efficiency. Reflow uses a pixel-based strategy to work with existing applications, and improves touch efficiency while minimally disrupting the design intent of the original application. Our system optimizes a UI by (i) extracting its layout from its screenshot, (ii) refining its layout, and (iii) re-rendering the UI to reflect these modifications. We conducted a user study with 10 participants and a heuristic evaluation with 6 experts and found that applications optimized by Reflow led to, on average, 9% faster selection time with minimal layout disruption. The results demonstrate that Reflow's refinements useful UI adaptations to improve touch interactions.
TokenMix: Rethinking Image Mixing for Data Augmentation in Vision Transformers

CutMix is a popular augmentation technique commonly used for training modern convolutional and transformer vision networks. It was originally designed to encourage Convolution Neural Networks (CNNs) to focus more on an image's global context instead of local information, which greatly improves the performance of CNNs. However, we found it to have limited benefits for transformer-based architectures that naturally have a global receptive field. In this paper, we propose a novel data augmentation technique TokenMix to improve the performance of vision transformers. TokenMix mixes two images at token level via partitioning the mixing region into multiple separated parts. Besides, we show that the mixed learning target in CutMix, a linear combination of a pair of the ground truth labels, might be inaccurate and sometimes counter-intuitive. To obtain a more suitable target, we propose to assign the target score according to the content-based neural activation maps of the two images from a pre-trained teacher model, which does not need to have high performance. With plenty of experiments on various vision transformer architectures, we show that our proposed TokenMix helps vision transformers focus on the foreground area to infer the classes and enhances their robustness to occlusion, with consistent performance gains. Notably, we improve DeiT-T/S/B with +1% ImageNet top-1 accuracy. Besides, TokenMix enjoys longer training, which achieves 81.2% top-1 accuracy on ImageNet with DeiT-S trained for 400 epochs. Code is available at https://github.com/Sense-X/TokenMix.
Three-dimensional numerical simulations of structured GRB jets

After the detection of GRB 170817A, the first unambiguous off-axis gamma-ray burst (GRB), several studies tried to understand the structure of GRB jets. The initial jet structure (directly produced by the central engine) can be partially preserved, or can be completely modified by the interaction with the environment. In this study, we perform three-dimensional, special relativistic hydrodynamics simulations of long GRB jets evolving through a massive progenitor star. Different jet scenarios were considered: Top-hat, Gaussian jets dominated by pressure or by kinetic energy, as well as a model of a supernova (SN) plus a jet both propagating through the progenitor. We found that, while propagating inside the progenitor star, jets with different initial structures are nearly indistinguishable. Kinetic dominated jets are faster and more collimated than pressure dominated jets. The dynamics of jets inside the progenitor star strongly depends on the presence of an associated SN, which can substantially decelerate the jet propagation. We show that the initial structure of GRB jets is preserved, or not, mainly depending on the jet collimation. The initial structure is preserved in uncollimated jets, i.e. jets which move through low density environments. Meanwhile, jets which move through dense environments are shaped by the interaction with the medium and remain collimated.
Spatial disconnection between stellar and dust emissions: The test of the Antennae Galaxies (Arp 244)

Context. The detection of dust-rich high-redshift galaxies with the Atacama Large Millimeter Array (ALMA), the cold dust emission of which is spatially disconnected from the ultraviolet emission, bears a challenge for modelling their spectral energy distributions (SEDs) with codes based on an energy budget between the stellar and dust components. Aims. We want to test the validity of energy balance modelling on a nearby resolved galaxy with vastly di ﬀ erent ultraviolet and infrared spatial distributions and to decipher the kinds of information that can be reliably retrieved from the analysis of the full SED. Methods. We used 15 broadband images of the Antennae Galaxies ranging from far-ultraviolet to far-infrared and divided Arp 244 into 58 square ∼ 1kpc 2 regions. We ﬁt the data with CIGALE to determine the star formation rate, stellar mass, and dust attenuation of each region. We compared these quantities to those obtained for Arp 244 as a whole. Results. The SEDs of the 58 regions and Arp 244 are well ﬁtted. The estimates of the star formation rate and stellar mass for the whole galaxy are found to be consistent – within one sigma – with the same parameters summed over the 58 regions. We present the spatial distribution of these physical parameters as well as the shape of the attenuation curve across the Antennae Galaxies. We ﬁnd that the Overlap Region exhibits a high star formation rate, attenuation, and a shallow attenuation curve. We observe a ﬂattening of the attenuation curves with increasing attenuation and dust surface density in agreement with the predictions of hydrodynamical simulations coupled with radiative transfer modelling.
Inspector: Pixel-Based Automated Game Testing via Exploration, Detection, and Investigation

Deep reinforcement learning (DRL) has attracted much attention in automated game testing. Early attempts rely on game internal information for game space exploration, thus requiring deep integration with games, which is inconvenient for practical applications. In this work, we propose using only screenshots/pixels as input for automated game testing and build a general game testing agent, Inspector, that can be easily applied to different games without deep integration with games. In addition to covering all game space for testing, our agent also tries to take human-like behaviors to interact with key objects in a game, since some bugs usually happen in player-object interactions. Inspector is based on purely pixel inputs and comprises three key modules: game space explorer, key object detector, and human-like object investigator. Game space explorer aims to explore the whole game space by using a curiosity-based reward function with pixel inputs. Key object detector aims to detect key objects in a game, based on a small number of labeled screenshots. Human-like object investigator aims to mimic human behaviors for investigating key objects via imitation learning. We conduct experiments on two popular video games: Shooter Game and Action RPG Game1. Experiment results demonstrate the effectiveness of Inspector in exploring game space, detecting key objects, and investigating objects. Moreover, Inspector successfully discovers two potential bugs in those two games. The demo video of Inspector is available at https://github.com/Inspector-GameTestinyInspector-GameTesting.
Single MR Image Super-Resolution using Generative Adversarial Network

Spatial resolution of medical images can be improved using super-resolution methods. Real Enhanced Super Resolution Generative Adversarial Network (Real-ESRGAN) is one of the recent effective approaches utilized to produce higher resolution images, given input images of lower resolution. In this paper, we apply this method to enhance the spatial resolution of 2D MR images. In our proposed approach, we slightly modify the structure of the Real-ESRGAN to train 2D Magnetic Resonance images (MRI) taken from the Brain Tumor Segmentation Challenge (BraTS) 2018 dataset. The obtained results are validated qualitatively and quantitatively by computing SSIM (Structural Similarity Index Measure), NRMSE (Normalized Root Mean Square Error), MAE (Mean Absolute Error) and VIF (Visual Information Fidelity) values.
JPerceiver: Joint Perception Network for Depth, Pose and Layout Estimation in Driving Scenes

Depth estimation, visual odometry (VO), and bird's-eye-view (BEV) scene layout estimation present three critical tasks for driving scene perception, which is fundamental for motion planning and navigation in autonomous driving. Though they are complementary to each other, prior works usually focus on each individual task and rarely deal with all three tasks together. A naive way is to accomplish them independently in a sequential or parallel manner, but there are many drawbacks, i.e., 1) the depth and VO results suffer from the inherent scale ambiguity issue; 2) the BEV layout is directly predicted from the front-view image without using any depth-related information, although the depth map contains useful geometry clues for inferring scene layouts. In this paper, we address these issues by proposing a novel joint perception framework named JPerceiver, which can simultaneously estimate scale-aware depth and VO as well as BEV layout from a monocular video sequence. It exploits the cross-view geometric transformation (CGT) to propagate the absolute scale from the road layout to depth and VO based on a carefully-designed scale loss. Meanwhile, a cross-view and cross-modal transfer (CCT) module is devised to leverage the depth clues for reasoning road and vehicle layout through an attention mechanism. JPerceiver can be trained in an end-to-end multi-task learning way, where the CGT scale loss and CCT module promote inter-task knowledge transfer to benefit feature learning of each task. Experiments on Argoverse, Nuscenes and KITTI show the superiority of JPerceiver over existing methods on all the above three tasks in terms of accuracy, model size, and inference speed. The code and models are available at~\href{https://github.com/sunnyHelen/JPerceiver}{https://github.com/sunnyHelen/JPerceiver}.
Robust AI Driving Strategy for Autonomous Vehicles

There has been significant progress in sensing, perception, and localization for automated driving, However, due to the wide spectrum of traffic/road structure scenarios and the long tail distribution of human driver behavior, it has remained an open challenge for an intelligent vehicle to always know how to make and execute the best decision on road given available sensing / perception / localization information. In this chapter, we talk about how artificial intelligence and more specifically, reinforcement learning, can take advantage of operational knowledge and safety reflex to make strategical and tactical decisions. We discuss some challenging problems related to the robustness of reinforcement learning solutions and their implications to the practical design of driving strategies for autonomous vehicles. We focus on automated driving on highway and the integration of reinforcement learning, vehicle motion control, and control barrier function, leading to a robust AI driving strategy that can learn and adapt safely.
Reverse Engineering Models of Concurrent Communicating Systems From Event Logs

—This paper tackles the problem of recovering formal models of communicating systems made up of components concurrently interacting with each other, e.g., Web service compositions or IoT systems. We present a passive model learning approach, which recovers, from event logs, one Input Output Labelled Transition System (IOLTS) for every component of the system under learning. From an event log, our approach is able to automatically recover conversations (a.k.a. sessions), without having any knowledge about the used event correlation mechanisms. It uses correlation pattern deﬁnitions and a heuristic based on the quality of the generated conversations to get the most relevant conversation sets. Then, our approach extracts the trace sets of every component and generates IOLTSs. The latter can be used as documentation, for test case generation, or for formal veriﬁcation.