Miracle-dz/UniFAR-FTUs huggingface dataset:
SAViT: Structure-Aware Vision Transformer Pruning via Collaborative Optimization

Vision Transformers (ViTs) yield impressive performance across various vision tasks. However, heavy computation and memory footprint make them inaccessible for edge devices. Previous works apply importance criteria determined independently by each individual component to prune ViTs. Considering that heterogeneous components in ViTs play distinct roles, these approaches lead to suboptimal performance. In this paper, we introduce joint importance, which integrates essential structural-aware interactions between components for the first time, to perform collaborative pruning. Based on the theoretical analysis, we construct a Taylor-based approximation to evaluate the joint importance. This guides pruning toward a more balanced reduction across all components. To further reduce the algorithm complexity, we incorporate the interactions into the optimization function under some mild assumptions. Moreover, the proposed method can be seamlessly applied to various tasks including object detection. Extensive experiments demonstrate the effectiveness of our method. Notably, the proposed approach outperforms the existing state-of-the-art approaches on ImageNet, increasing accuracy by 0.7% over the DeiT-Base baseline while saving 50% FLOPs. On COCO, we are the first to show that 70% FLOPs of Faster R-CNN with ViT backbone can be removed with only 0.3% mAP drop. The code is available at https://github.com/hikvision-research/SAViT .
Finding Correlated Equilibrium of Constrained Markov Game: A Primal-Dual Approach

Constrained Markov game is a fundamental problem that covers many applications, where multiple agents compete with each other under behavioral constraints. The existing literature has proved the existence of Nash equilibrium for constrained Markov games, which turns out to be PPAD-complete and cannot be computed in polynomial time. In this work, we propose a surrogate notion of correlated equilibrium (CE) for constrained Markov games that can be computed in polynomial time, and study its fundamental properties. We show that the modification structure of CE of constrained Markov games is fundamentally different from that of unconstrained Markov games. Moreover, we prove that the corresponding Lagrangian function has zero duality gap. Based on these results, we develop the first primal-dual algorithm that provably converges to CE of constrained Markov games. In particular, we prove that both the duality gap and the constraint violation of the output policy converge at the rate O ( 1 √ T ) . Moreover, when adopting the V-learning algorithm as the subroutine in the primal update, our algorithm achieves an approximate CE with ϵ duality gap with the sample complexity O ( H 9 SA 2 ϵ − 4 ) .
C2FAR: Coarse-to-Fine Autoregressive Networks for Precise Probabilistic Forecasting

We present coarse-to-fine autoregressive networks (C2FAR), a method for modeling the probability distribution of univariate, numeric random variables. C2FAR generates a hierarchical, coarse-to-fine discretization of a variable autoregressively; progressively finer intervals of support are generated from a sequence of binned distributions, where each distribution is conditioned on previously-generated coarser intervals. Unlike prior (flat) binned distributions, C2FAR can represent values with exponentially higher precision, for only a linear increase in complexity. We use C2FAR for probabilistic forecasting via a recurrent neural network, thus modeling time series autoregressively in both space and time. C2FAR is the first method to simultaneously handle discrete and continuous series of arbitrary scale and distribution shape. This flexibility enables a variety of time series use cases, including anomaly detection, interpolation, and compression. C2FAR achieves improvements over the state-of-the-art on several benchmark forecasting datasets.
Pre-Trained Model Reusability Evaluation for Small-Data Transfer Learning

We study model reusability evaluation (MRE) for source pre-trained models: evaluating their transfer learning performance to new target tasks. In special, we focus on the setting under which the target training datasets are small, making it difﬁ-cult to produce reliable MRE scores using them. Under this situation, we propose synergistic learning for building the task-model metric, which can be realized by collecting a set of pre-trained models and asking a group of data providers to participate. We provide theoretical guarantees to show that the learned task-model metric distances can serve as trustworthy MRE scores, and propose synergistic learning algorithms and models for general learning tasks. Experiments show that the MRE models learned by synergistic learning can generate signiﬁcantly more reliable MRE scores than existing approaches for small-data transfer learning.
AZ-whiteness test: a test for signal uncorrelation on spatio-temporal graphs

We present the ﬁrst whiteness hypothesis test for graphs, i.e., a whiteness test for multivariate time series associated with the nodes of a dynamic graph; as such, the test represents an important model assessment tool for graph deep learning, e.g., in forecasting setups. The statistical test aims at detecting existing serial dependencies among close-in-time observations, as well as spatial dependencies among neighboring observations given the underlying graph. The proposed AZ-test can be intended as a spatio-temporal extension of traditional tests designed for system identiﬁcation to graph signals. The AZ-test is versatile, allowing the underlying graph to be dynamic, changing in topology and set of nodes over time, and weighted, thus accounting for connections of different strength, as it is the case in many application scenarios like sensor and transportation networks. The asymptotic distribution of the designed test can be derived under the null hypothesis without assuming identically distributed data. We show the effectiveness of the test on both synthetic and real-world problems, and illustrate how it can be employed to assess the quality of spatio-temporal forecasting models by analyzing the prediction residuals appended to the graph stream.
Improving Generative Adversarial Networks via Adversarial Learning in Latent Space

For Generative Adversarial Networks which map a latent distribution to the target distribution, in this paper, we study how the sampling in latent space can affect the generation performance, especially for images. We observe that, as the neural generator is a continuous function, two close samples in latent space would be mapped into two nearby images, while their quality can differ much as the quality generally does not exhibit a continuous nature in pixel space. From such a continuous mapping function perspective, it is also possible that two distant latent samples can be mapped into two close images (if not exactly the same). In particular, if the latent samples are mapped in aggregation into a single mode, mode collapse occurs. Accordingly, we propose adding an implicit latent transform before the mapping function to improve latent z from its initial distribution, e.g., Gaussian. This is achieved using well-developed adversarial sample mining techniques, e.g. iterative fast gradient sign method (I-FGSM). We further propose new GAN training pipelines to obtain better generative mappings w.r. t quality and diversity by introducing targeted latent transforms into the bi-level optimization of GAN. Experimental results on visual data show that our method can effectively achieve improvement in both quality and diversity. The implementation is publicly available at https://github.com/yangco-le/AdvLatGAN .
Enhanced Latent Space Blind Model for Real Image Denoising via Alternative Optimization

Motivated by the achievements in model-based methods and the advances in deep networks, we propose a novel enhanced latent space blind model based deep unfolding network, namely ScaoedNet, for complex real image denoising. It is derived by introducing latent space, noise information, and guidance constraint into the denoising cost function. A self-correction alternative optimization algorithm is proposed to split the novel cost function into three alternative subproblems, i.e. , guidance representation (GR), degradation estimation (DE) and reconstruction (RE) subproblems. Finally, we implement the optimization process by a deep unfolding network consisting of GR, DE and RE networks. For higher performance of the DE network, a novel parameter-free noise feature adaptive enhancement (NFAE) layer is proposed. To synchronously and dynamically realize internal-external feature information mining in the RE network, a novel feature multi-modulation attention (FM 2 A) module is proposed. Our approach thereby leverages the advantages of deep learning, while also beneﬁting from the principled denoising provided by the classical model-based formulation. To the best of our knowledge, our enhanced latent space blind model, optimization scheme, NFAE and FM 2 A have not been reported in the previous literature. Experimental results show the promising performance of ScaoedNet on real image denoising. Code is available at https://github.com/chaoren88/ScaoedNet .
Coreset for Line-Sets Clustering

The input to the line-sets k -median problem is an integer k ≥ 1 , and a set L = { L 1 , . , L n } that contains n sets of lines in R d . The goal is to compute a set C of k centers (points in R d ) that minimizes the sum (cid:80) L ∈L min ℓ ∈ L,c ∈ C dist( ℓ, c ) of Euclidean distances from each set to its closest center, where dist( ℓ, c ) := min x ∈ ℓ ∥ x − c ∥ 2 . An ε -coreset for this problem is a weighted subset of sets in L that approximates this sum up to 1 ± ε multiplicative factor, for every set C of k centers. We prove that every such input set L has a small ε -coreset, and provide the first coreset construction for this problem and its variants. The coreset consists of O (log 2 n ) weighted line-sets from L , and is constructed in O ( n log n ) time for every fixed d, k ≥ 1 and ε ∈ (0 , 1) . The main technique is based on a novel reduction to a “fair clustering” of colored points to colored centers. We then provide a coreset for this coloring problem, which may be of independent interest. Open source code and experiments are also provided.
Kinetic Alfven waves and plasma transport at the magnetopause

Large amplitude compressional type waves, with frequencies ranging from 10--500 mHz, are nearly always found in the magnetosheath near the magnetopause where there are large gradients in density, pressure and magnetic field. As compressional waves propagation to the magnetopause, there gradients efficiently couple them with shear/kinetic Alfven waves near the Alfven field-line resonance location ({omega} = k{sub {parallel}} v{sub A}). The authors present a solution of the kinetic-MHD wave equations for this process using a realistic equilibrium profile including full ion Larmor radius effects and wave-particle resonance interactions for electrons and ions to model the dissipation. For northward IMF a KAW propagates backward to the magnetosheath. For southward IMF the wave remains in the magnetopause but can propagate through the k{sub {parallel}} = 0 location. The quasi-linear theory predicts that KAWs produce plasma transport with a diffusion coefficient D{sub {perpendicular}} {approximately} 10{sup 9} m{sup 2}/s and plasma convection on the order of 1 km/s. However, for southward IMF additional transport can occur because magnetic islands form at the k{sub {parallel}} = 0 location. Due to the broadband nature of the observed waves these islands can overlap leading to stochastic transport which is much larger than that due to quasilinear effects.
Generalized Laplacian Eigenmaps

Graph contrastive learning attracts/disperses node representations for similar/dissimilar node pairs under some notion of similarity. It may be combined with a low-dimensional embedding of nodes to preserve intrinsic and structural properties of a graph. COLES, a recent graph contrastive method combines traditional graph embedding and negative sampling into one framework. COLES in fact minimizes the trace difference between the within-class scatter matrix encapsulating the graph connectivity and the total scatter matrix encapsulating negative sampling. In this paper, we propose a more essential framework for graph embedding, called Generalized Laplacian EigeNmaps (GLEN), which learns a graph representation by maximizing the rank difference between the total scatter matrix and the within-class scatter matrix, resulting in the minimum class separation guarantee. However, the rank difference minimization is an NP-hard problem. Thus, we replace the trace difference that corresponds to the difference of nuclear norms by the difference of LogDet expressions, which we argue is a more accurate surrogate for the NP-hard rank difference than the trace difference. While enjoying a lesser computational cost, the difference of LogDet terms is lower-bounded by the Affine-invariant Riemannian metric (AIRM) and upper-bounded by AIRM scaled by the factor of √ m . We show on popular benchmarks/backbones that GLEN offers favourable accuracy/scalability compared to state-of-the-art baselines.
DropCov: A Simple yet Effective Method for Improving Deep Architectures

Previous works show global covariance pooling (GCP) has great potential to improve deep architectures especially on visual recognition tasks, where post-normalization of GCP plays a very important role in ﬁnal performance. Although several post-normalization strategies have been studied, these methods pay more close attention to effect of normalization on covariance representations rather than the whole GCP networks, and their effectiveness requires further understanding. Meanwhile, existing effective post-normalization strategies (e.g., matrix power normalization) usually suffer from high computational complexity (e.g., O ( d 3 ) for d -dimensional inputs). To handle above issues, this work ﬁrst analyzes the effect of post-normalization from the perspective of training GCP networks. Particularly, we for the ﬁrst time show that effective post-normalization can make a good trade-off between representation decorrelation and information preservation for GCP, which are crucial to alleviate over-ﬁtting and increase representation ability of deep GCP networks, respectively . Based on this ﬁnding, we can improve existing post-normalization methods with some small modiﬁcations, providing further support to our observation. Furthermore, this ﬁnding encourages us to propose a novel pre-normalization method for GCP (namely DropCov), which develops an adaptive channel dropout on features right before GCP, aiming to reach trade-off between representation decorrelation and information preservation in a more efﬁcient way. Our DropCov only has a linear complexity of O ( d ) , while being free for inference.
Towards Video Text Visual Question Answering: Benchmark and Baseline

There are already some text-based visual question answering (TextVQA) benchmarks for developing machine’s ability to answer questions based on texts in images in recent years. However, models developed on these benchmarks cannot work effectively in many real-life scenarios (e.g. traffic monitoring, shopping ads and e-learning videos) where temporal reasoning ability is required. To this end, we propose a new task named Vi deo Te xt V isual Q uestion A nswering (ViteVQA in short) that aims at answering questions by spatiotemporally reasoning texts and visual information in a given video. In particular, on the one hand, we build the first ViteVQA benchmark dataset named M4-ViteVQA — the abbreviation of M ulti-category M ulti-frame M ulti-resolution M ulti-modal benchmark for ViteVQA , which contains 7,620 video clips of 9 categories (i.e., shopping , traveling , driving , vlog , sport , advertisement , movie , game and talking ) and 3 kinds of resolutions (i.e., 720p, 1080p and 1176 × 664), and 25,123 question-answer pairs. On the other hand, we develop a baseline method named T5-ViteVQA for the ViteVQA task. T5-ViteVQA consists of five transformers. It first extracts optical character recognition (OCR) tokens, question features, and video representations via two OCR transformers, one language transformer and one video-language transformer, respectively. Then, a multimodal fusion transformer and an answer generation module are applied to fuse multimodal information and generate the final prediction. Extensive experiments on M4-ViteVQA demonstrate the superiority of T5-ViteVQA over the existing approaches of TextVQA and VQA tasks. The ViteVQA benchmark is available in https://github.com/bytedance/VTVQA.
Out-of-Distribution Detection with An Adaptive Likelihood Ratio on Informative Hierarchical VAE

Unsupervised out-of-distribution (OOD) detection is essential for the reliability of machine learning. In the literature, existing work has shown that higher-level semantics captured by hierarchical VAEs can be used to detect OOD instances. However, we empirically show that, the inheirt “ posterior collapse ” of hierarchical VAEs would seriously limit their capacity for OOD detection. Based on a thorough analysis, we propose an informative hierarchical VAE to alleviate this issue through enhancing the connections between the data sample and its multi-layer stochastic latent representations during training. Furthermore, we propose a novel score function for unsupervised OOD detection, referred to as Adaptive Likelihood Ratio, which can selectively aggregate the semantic information on multiple hidden layers of hierarchical VAEs, leading to a strong separability between in-distribution and OOD samples. Experimental results demonstrate that our method can significantly outperform existing state-of-the-art unsupervised OOD detection approaches.
Multivariate Time-Series Forecasting with Temporal Polynomial Graph Neural Networks

Modeling multivariate time series (MTS) is critical in modern intelligent systems. The accurate forecast of MTS data is still challenging due to the complicated latent variable correlation. Recent works apply the Graph Neural Networks (GNNs) to the task, with the basic idea of representing the correlation as a static graph. However, predicting with a static graph causes signiﬁcant bias because the correlation is time-varying in the real-world MTS data. Besides, there is no gap analysis between the actual correlation and the learned one in their works to validate the effectiveness. This paper proposes a temporal polynomial graph neural network (TPGNN) for accurate MTS forecasting, which represents the dynamic variable correlation as a temporal matrix polynomial in two steps. First, we capture the overall correlation with a static matrix basis. Then, we use a set of time-varying coefﬁcients and the matrix basis to construct a matrix polynomial for each time step. The constructed result empirically captures the precise dynamic correlation of six synthetic MTS datasets generated by a non-repeating random walk model. Moreover, the theoretical analysis shows that TPGNN can achieve perfect approximation under a commutative condition. We conduct extensive experiments on two trafﬁc datasets with prior structure and four benchmark datasets. The results indicate that TPGNN achieves the state-of-the-art on both short-term and long-term MTS forecastings. 1
Scientific Explanation: Putting Communication First

Scientific explanations must bear the proper relationship to the world: they must depict what, out in the world, is responsible for the explanandum. But explanations must also bear the proper relationship to their audience: they must be able to create human understanding. With few exceptions, philosophical accounts of explanation either ignore entirely the relationship between explanations and their audience or else demote this consideration to an ancillary role. In contrast, I argue that considering an explanation’s communicative role is crucial to any satisfactory account of explanation.
MECHANISMS FOR MOTION PERCEPTION

ne of the goals of vision science is to understand the workings of visual systems from the diverse perspectives of psychophysics, neurophysiol-ogy, and computation, and to develop this understanding into an integrated theory of vision. This program has been particularly successful in the study of motion perception. Let us begin by considering one of the long-standing problems of motion perception: the phenomenon of apparent motion, by which a rapid sequence of movie frames gives rise to the impression of motion. For many years, this was a hotly debated phenomenon. But recently a unified view has emerged that combines insights from several converging lines of investigation. In this view, motion is conceptualized as a kind of slant or orientation in space-time. Any system that is designed to detect this space-time orientation will inevitably experience the "illusion" of apparent motion in the right conditions From this perspective, it is no mystery that we see motion in movies, and it is probably misleading to think of the effect as an illusion. To illustrate the spatio-temporal approach, consider a scene containing a moving object, such as that shown in Figure 1 (adapted from Ref. 1). In Figure 1a, a vase is seen to move in a rightward direction. A movie of this scene would consist of a sequence of vase images in which the vase is successively displaced to the right. We could stack all of these frames up into a solid, as illustrated in Figure 1b, making something like a flip-book; a skeleton view is shown in Figure 1c. We are using the third dimension to represent time, and since images contain only the two spatial dimensions (x,y), the axes of our spatio-temporal representations are (x,y,t). FIGURE 1. a: An image of a vase moving to the right. b: A sequence of frames may be piled up to form a flip book; time is the third dimension. c: A skeleton view of the (x,y,t) volume helps suggest its structure. d: The space-time volume may be sliced to illustrate the fact that the motion is equivalent to spatio-temporal orientation. e: In the case of continuous motion, the volume is densely filled. The moving vase traces out an extruded shape that is sheared due to the motion. O (a) (b) (c) (d) (e) y x t y x t y x t y x t
On the Instability of Ekman Boundary Flow

Abstract The stability of the two-dimensional boundary flow produced in a rotating tank with small inflow is investigated by means of perturbation analysis. The resulting differential eigenvalue problem is solved numerically both for the complete set of equations and for a truncated set (the Orr-Sommerfeld equation) shown by Stuart and Barcilon to be valid in the limit of large Reynolds number R. Both solutions exhibit instability, with a critical R of about 55 for the complete solution and 93 for the truncated set, where R is based on the boundary scale depth and the relative velocity of the fluid distant from the boundary. Stationary waves become unstable for R > 115. These values are to be compared with Faller's observed value of R ≈ 125 for nearly stationary waves. The difference between theory and observation is discussed and largely rationalized as being due to the difficulty of observing moving waves with Faller's method. The effect of the terms neglected in Stuart's analysis is shown to lead to a .