Miracle-dz/UniFAR-FTUs huggingface dataset:
Multimodal learning-based inversion models for the space-time reconstruction of satellite-derived geophysical fields

For numerous earth observation applications, one may benefit from various satellite sensors to address the reconstruction of some process or information of interest. A variety of satellite sensors deliver observation data with different sampling patterns due satellite orbits and/or their sensitivity to atmospheric conditions (e.g., clour cover, heavy rains,.). Beyond the ability to account for irregularly-sampled observations, the definition of model-driven inversion methods is often limited to specific case-studies where one can explicitly derive a physical model to relate the different observation sources. Here, we investigate how end-to-end learning schemes provide new means to address multimodal inversion problems. The proposed scheme combines a variational formulation with trainable observation operators, {\em a priori} terms and solvers. Through an application to space oceanography, we show how this scheme can successfully extract relevant information from satellite-derived sea surface temperature images and enhance the reconstruction of sea surface currents issued from satellite altimetry data.
ESS: Learning Event-based Semantic Segmentation from Still Images

Retrieving accurate semantic information in challenging high dynamic range (HDR) and high-speed conditions remains an open challenge for image-based algorithms due to severe image degradations. Event cameras promise to address these challenges since they feature a much higher dynamic range and are resilient to motion blur. Nonetheless, semantic segmentation with event cameras is still in its infancy which is chiefly due to the lack of high-quality, labeled datasets. In this work, we introduce ESS (Event-based Semantic Segmentation), which tackles this problem by directly transferring the semantic segmentation task from existing labeled image datasets to unlabeled events via unsupervised domain adaptation (UDA). Compared to existing UDA methods, our approach aligns recurrent, motion-invariant event embeddings with image embeddings. For this reason, our method neither requires video data nor per-pixel alignment between images and events and, crucially, does not need to hallucinate motion from still images. Additionally, we introduce DSEC-Semantic, the first large-scale event-based dataset with fine-grained labels. We show that using image labels alone, ESS outperforms existing UDA approaches, and when combined with event labels, it even outperforms state-of-the-art supervised approaches on both DDD17 and DSEC-Semantic. Finally, ESS is general-purpose, which unlocks the vast amount of existing labeled image datasets and paves the way for new and exciting research directions in new fields previously inaccessible for event cameras.
DirecFormer: A Directed Attention in Transformer Approach to Robust Action Recognition

Human action recognition has recently become one of the popular research topics in the computer vision community. Various 3D-CNN based methods have been presented to tackle both the spatial and temporal dimensions in the task of video action recognition with competitive results. However, these methods have suffered some fundamental limitations such as lack of robustness and generalization, e.g., how does the temporal ordering of video frames affect the recognition results? This work presents a novel end-to-end Transformer-based Directed Attention (Direc-Former) framework11The implementation of DirecFormer is available at https://github.com/uark-cviu/DirecFormer for robust action recognition. The method takes a simple but novel perspective of Transformer-based approach to understand the right order of sequence actions. Therefore, the contributions of this work are three-fold. Firstly, we introduce the problem of ordered temporal learning issues to the action recognition problem. Secondly, a new Directed Attention mechanism is introduced to understand and provide attentions to human actions in the right order. Thirdly, we introduce the conditional dependency in action sequence modeling that includes orders and classes. The proposed approach consistently achieves the state-of-the-art (SOTA) results compared with the recent action recognition methods [4, 18, 72, 74]. on three standard large-scale benchmarks, i.e. Jester, Kinetics-400 and Something-Something-V2.
A note on the distribution of pinned distance trees in dense sets

The recent breakthrough result of Guth, Iosevich, Ou, and Wang (2019) on the Falconer distance problem states that for a compact set A ⊂ R 2 , if the Hausdorﬀ dimension of A is greater than 54 , then the distance set ∆( A ) has positive Lebesgue measure. Ou and Taylor (2021) recently generalized this result to the setting of (distance) trees. The main purpose of this paper is to study the discrete version of their result over both prime ﬁelds and arbitrary ﬁelds for small sets.
OWC-enabled Spine and Leaf Architecture Towards Energy Efficient Data Center Networks

Due to the emergence of new paradigms and services such as 5G/6G, IoT, and more, current deployed wired Data Center Networks (DCNs) are not meeting the required performance metrics due to their limited reconfigurability, scalability, and throughput. To that end, wireless DCNs using technologies such as Optical Wireless Communication (OWC) have become viable and cost-effective solutions as they offer higher capacity, better energy efficiency, and better scalability. This paper proposes an OWC-based spine and leafDCNs where the leaf switches are enabled with OWC transceivers, and the spine switches are replaced by Access Points (APs) in the ceiling connected to a backbone network. The APs are interconnected through a Passive Optical Network (PON) that also connects the architecture with upper network layers. An Infrared (IR) OWC system that employs Wavelength Division Multiplexing (WDM) is proposed to enhance the DCN downlink communication. The simulation (i.e., channel modeling) results show that our proposed data center links achieve good data rates in the data center up to 15 Gbps. For the PON, Arrayed Waveguide Grating Routers (AWGRs) that enable WDM are proposed to connect the APs. We evaluate the performance of the considered architecture in term of its power efficiency compared to traditional spine and leaf data centers. The results show that the OWC-enabled DCN reduces the power consumption by 42% compared to traditional the spine and leaf architecture.
Learning When to Translate for Streaming Speech

How to find proper moments to generate partial sentence translation given a streaming speech input? Existing approaches waiting-and-translating for a fixed duration often break the acoustic units in speech, since the boundaries between acoustic units in speech are not even. In this paper, we propose MoSST, a simple yet effective method for translating streaming speech content. Given a usually long speech sequence, we develop an efficient monotonic segmentation module inside an encoder-decoder model to accumulate acoustic information incrementally and detect proper speech unit boundaries for the input in speech translation task. Experiments on multiple translation directions of the MuST-C dataset show that outperforms existing methods and achieves the best trade-off between translation quality (BLEU) and latency. Our code is available at https://github.com/dqqcasia/mosst.
Feature Selection for Vertex Discriminant Analysis

We revisit vertex discriminant analysis (VDA) from the perspective of proximal distance algorithms. By specifying sparsity sets as constraints that directly control the number of active features, VDA is able to fit multiclass classifiers with no more than k active features. We combine our sparse VDA approach with repeated cross validation to fit classifiers across the full range of model sizes on a given dataset. Our numerical examples demonstrate that grappling with sparsity directly is an attractive approach to model building in high-dimensional settings. Applications to kernel-based VDA are also considered. Support vector machines (SVM) stand out in the universe of classification algorithms for their simplicity, interpretability, speed, and theoretical guarantees (Smola and Schölkopf, 1998; Vapnik, 1999). The circle of SVM algorithms have been bolstered by fundamental advances in statistical theory and computer hardware but also challenged by the growing size of modern datasets. The Achilles heal of SVMs is their limitation to binary classification. Strategies to overcome this limitation, most notably one-versus-rest and one-versus-one, rely on multiple SVMs to discriminate between classes. This necessarily increases time to fit a model, obscures statistical interpretation, and violates the principle of parsimony. Vertex discriminant analysis (VDA) (Lange and Wu, 2008) addresses this particular weakness of SVMs. Our subsequent paper (Wu and Lange, 2010) derives a fast cyclic coordinate descent algorithm and attacks feature selection by a combination of Euclidean group and lasso penalties. Our second followup paper (Wu and Wu, 2012) generalizes VDA to nonlinear discriminant analysis by the kernel trick. The current paper revisits the problem of feature selection in VDA from the perspective of distance to set penalties. This allows us to incorporate a spectrum of constraints, with particular emphasis on sparsity sets. The experiments recorded here demonstrate that the direct control of sparsity is highly effective in building parsimonious models on both simulated and real-world datasets. ar X iv :2 20 3. 11 16 8v 1 [ st at . C O ] 2 1 M ar 2 02 2 Feature Selection for Vertex Discriminant Analysis A PREPRINT
MatchFormer: Interleaving Attention in Transformers for Feature Matching

Local feature matching is a computationally intensive task at the subpixel level. While detector-based methods coupled with feature descriptors struggle in low-texture scenes, CNN-based methods with a sequential extract-to-match pipeline, fail to make use of the matching capacity of the encoder and tend to overburden the decoder for matching. In contrast, we propose a novel hierarchical extract-and-match transformer, termed as MatchFormer. Inside each stage of the hierarchical encoder, we interleave self-attention for feature extraction and cross-attention for feature matching, yielding a human-intuitive extract-and-match scheme. Such a match-aware encoder releases the overloaded decoder and makes the model highly efficient. Further, combining self- and cross-attention on multi-scale features in a hierarchical architecture improves matching robustness, particularly in low-texture indoor scenes or with less outdoor training data. Thanks to such a strategy, MatchFormer is a multi-win solution in efficiency, robustness, and precision. Compared to the previous best method in indoor pose estimation, our lite MatchFormer has only 45% GFLOPs, yet achieves a +1.3% precision gain and a 41% running speed boost. The large MatchFormer reaches state-of-the-art on four different benchmarks, including indoor pose estimation (ScanNet), outdoor pose estimation (MegaDepth), homography estimation and image matching (HPatch), and visual localization (InLoc).
A Dual-Arm Robot that Manipulates Heavy Plates Cooperatively with a Vacuum Lifter

A vacuum lifter is widely used to hold and pick up large, heavy, and flat objects. Conventionally, when using a vacuum lifter, a human worker watches the state of a running vacuum lifter and adjusts the object's pose to maintain balance. In this work, we propose using a dual-arm robot to replace the human workers and develop planning and control methods for a dual-arm robot to raise a heavy plate with the help of a vacuum lifter. The methods help the robot determine its actions by considering the vacuum lifer's suction position and suction force limits. The essence of the methods is two-fold. First, we build a Manipulation State Graph (MSG) to store the weighted logical relations of various plate contact states and robot/vacuum lifter configurations, and search the graph to plan efficient and low-cost robot manipulation sequences. Second, we develop a velocity-based impedance controller to coordinate the robot and the vacuum lifter when lifting an object. With its help, a robot can follow the vacuum lifter's motion and realize compliant robot-vacuum lifter collaboration. The proposed planning and control methods are investigated using real-world experiments. The results show that a robot can effectively and flexibly work together with a vacuum lifter to manipulate large and heavy plate-like objects with the methods' support.
Voltage-dependent synaptic plasticity: Unsupervised probabilistic Hebbian plasticity rule based on neurons membrane potential

This study proposes voltage-dependent-synaptic plasticity (VDSP), a novel brain-inspired unsupervised local learning rule for the online implementation of Hebb’s plasticity mechanism on neuromorphic hardware. The proposed VDSP learning rule updates the synaptic conductance on the spike of the postsynaptic neuron only, which reduces by a factor of two the number of updates with respect to standard spike timing dependent plasticity (STDP). This update is dependent on the membrane potential of the presynaptic neuron, which is readily available as part of neuron implementation and hence does not require additional memory for storage. Moreover, the update is also regularized on synaptic weight and prevents explosion or vanishing of weights on repeated stimulation. Rigorous mathematical analysis is performed to draw an equivalence between VDSP and STDP. To validate the system-level performance of VDSP, we train a single-layer spiking neural network (SNN) for the recognition of handwritten digits. We report 85.01 ± 0.76% (Mean ± SD) accuracy for a network of 100 output neurons on the MNIST dataset. The performance improves when scaling the network size (89.93 ± 0.41% for 400 output neurons, 90.56 ± 0.27 for 500 neurons), which validates the applicability of the proposed learning rule for spatial pattern recognition tasks. Future work will consider more complicated tasks. Interestingly, the learning rule better adapts than STDP to the frequency of input signal and does not require hand-tuning of hyperparameters.
Towards Structuring Real-World Data at Scale: Deep Learning for Extracting Key Oncology Information from Clinical Text with Patient-Level Supervision

Objective: The majority of detailed patient information in real-world data (RWD) is only consistently available in free-text clinical documents. Manual curation is expensive and time-consuming. Developing natural language processing (NLP) methods for structuring RWD is thus essential for scaling real-world evidence generation. Materials and Methods: Traditional rule-based systems are vulnerable to the prevalent linguistic variations and ambiguities in clinical text, and prior applications of machine-learning methods typically require sentence-level or report-level labeled examples that are hard to produce at scale. We propose leveraging patient-level supervision from medical registries, which are often readily available and capture key patient information, for general RWD applications. To combat the lack of sentence-level or report-level annotations, we explore advanced deep-learning methods by combining domain-specific pretraining, recurrent neural networks, and hierarchical attention. Results: We conduct an extensive study on 135,107 patients from the cancer registry of a large integrated delivery network (IDN) comprising healthcare systems in five western US states. Our deep learning methods attain test AUROC of 94-99% for key tumor attributes and comparable performance on held-out data from separate health systems and states. Discussion and Conclusion: Ablation results demonstrate clear superiority of these advanced deep-learning methods over prior approaches. Error analysis shows that our NLP system sometimes even corrects errors in registrar labels. We also conduct a preliminary investigation in accelerating registry curation and general RWD structuring via assisted curation for over 1.2 million cancer patients in this healthcare network.
The Dark Side: Security Concerns in Machine Learning for EDA

The growing IC complexity has led to a compelling need for design efficiency improvement through new electronic design automation (EDA) methodologies. In recent years, many unprecedented efficient EDA methods have been enabled by machine learning (ML) techniques. While ML demonstrates its great potential in circuit design, however, the dark side about security problems, is seldomly discussed. This paper gives a comprehensive and impartial summary of all security concerns we have observed in ML for EDA. Many of them are hidden or neglected by practitioners in this field. In this paper, we first provide our taxonomy to define four major types of security concerns, then we analyze different application scenarios and special properties in ML for EDA. After that, we present our detailed analysis of each security concern with experiments.
Generating Fast and Slow: Scene Decomposition via Reconstruction

We consider the problem of segmenting scenes into constituent entities, i.e. underlying objects and their parts. Current supervised visual detectors though impressive within their training distribution, often fail to segment out-of-distribution scenes into their constituent entities. Recent slot-centric generative models break such dependence on supervision, by attempting to segment scenes into entities unsupervised, by reconstructing pixels. However, they have been restricted thus far to toy scenes as they suffer from a reconstruction-segmentation trade-off: as the entity bottleneck gets wider, reconstruction improves but then the segmentation collapses. We propose GFS-Nets (Generating Fast and Slow Networks) that alleviate this issue with two ingredients: i) curriculum training in the form of primitives, often missing from current generative models and, ii) test-time adaptation per scene through gradient descent on the reconstruction objective, what we call slow inference, missing from current feed-forward detectors. We show the proposed curriculum suffices to break the reconstruction-segmentation trade-off, and slow inference greatly improves segmentation in out-of-distribution scenes. We evaluate GFS-Nets in 3D and 2D scene segmentation benchmarks of PartNet
A Learning Framework for Bandwidth-Efficient Distributed Inference in Wireless IoT

The limited bandwidth and power resources of wireless sensors in distributed environments have resulted in new challenges in handling the ever-growing volume of transmissions generated by the Internet-of-Things (IoT) applications. To overcome these challenges, each sensor should compress and quantize its observations before sending them to a fusion center (FC) for global decision inference. Unfortunately, most of the conventional compression techniques and entropy quantizers only focus on reconstruction fidelity as a performance measure, neglecting the sensing goal. In this article, we propose a joint design of compression mechanisms and entropy quantizers with the sensing goal of machine-to-machine (M2M) communications. We define a deep learning-based framework for compressing and quantizing observations from correlated sensors. Unlike traditional methods, our proposed method not only maximizes the reconstruction fidelity but also optimally compresses sensor observations in terms of the accuracy of the inferred decision (i.e., the sensing goal) at the FC. The proposed framework is widely applicable as it does not impose any assumptions on observation distribution. Furthermore, a novel loss function has been proposed to focus on learning complementary features at each sensor. Our experimental results demonstrate that the framework outperforms other benchmark models.
Smoothing algorithms for projected center-vortex gauge fields

We study the application of SU(3) gauge field smoothing methods to Z(3)-projected center-vortex gauge fields. Due to the proportionality of the vortex links to the identity, naive applications of these methods are either ineffectual or limited in scope, containing subtle issues which are not obviously manifest. To overcome these issues we introduce centrifuge preconditioning, a novel method applied prior to smoothing that rotates the links away from the center while preserving the fundamental structure of the vortex field. Additionally, the concept of vortex-preserved annealed smoothing is formulated to ensure the smoothing procedure maintains the underlying vortex structure. The application of these new methods in the context of annealed smoothing applied to vortex fields is shown to successfully achieve the desired smoothness condition required for the study of more advanced operators.
Determining the Beaming of Io Decametric Emissions: A Remote Diagnostic to Probe the Io‐Jupiter Interaction

We investigate the beaming of 11 Io‐Jupiter decametric (Io‐DAM) emissions observed by Juno/Waves, the Nançay Decameter Array, and NenuFAR. Using an up‐to‐date magnetic field model and three methods to position the active Io Flux Tube (IFT), we accurately locate the radiosources and determine their emission angle θ from the local magnetic field vector. These methods use (a) updated models of the IFT equatorial lead angle, (b) ultraviolet (UV) images of Jupiter's aurorae, and (c) multi‐point radio measurements. The kinetic energy Ee− of source electrons is then inferred from θ in the framework of the Cyclotron Maser Instability. The precise position of the active IFT achieved from methods (b and c) can be used to test the effective plasma density of the Io torus. Simultaneous radio/UV observations reveal that multiple Io‐DAM arcs are associated with multiple UV spots and provide the first direct evidence of an Io‐DAM arc associated with a trans‐hemispheric beam UV spot. Multi‐point radio observations probe the Io‐DAM sources at various altitudes, times and hemispheres. Overall, θ varies a function of frequency (altitude), by decreasing from 75°−80° to 70°−75° over 10−40 MHz with slightly larger values in the northern hemisphere, and independently varies as a function of time (or longitude of Io). Its uncertainty of a few degrees is dominated by the error on the longitude of the active IFT. The inferred values of Ee− also vary as a function of altitude and time. For the 11 investigated cases, they range from 3 to 16 keV, with a 6.6 ± 2.7 keV average.
An In-depth Comparative Analysis of Cloud Block Storage Workloads: Findings and Implications

Cloud block storage systems support diverse types of applications in modern cloud services. Characterizing their input/output (I/O) activities is critical for guiding better system designs and optimizations. In this article, we present an in-depth comparative analysis of production cloud block storage workloads through the block-level I/O traces of billions of I/O requests collected from two production systems, Alibaba Cloud and Tencent Cloud Block Storage. We study their characteristics of load intensities, spatial patterns, and temporal patterns. We also compare the cloud block storage workloads with the notable public block-level I/O workloads from the enterprise data centers at Microsoft Research Cambridge, and we identify the commonalities and differences of the three sources of traces. To this end, we provide 6 findings through the high-level analysis and 16 findings through the detailed analysis on load intensity, spatial patterns, and temporal patterns. We discuss the implications of our findings on load balancing, cache efficiency, and storage cluster management in cloud block storage systems.