Miracle-dz/UniFAR-FTUs huggingface dataset:
Learning Tactile Models for Factor Graph-based State Estimation

We address the problem of estimating object pose from touch during manipulation under occlusion. Vision-based tactile sensors provide rich, local measurements at the point of contact. A single such measurement, however, contains limited information and multiple measurements are needed to infer latent object state. We solve this inference problem using a factor graph. In order to incorporate tactile measurements in the graph, we need local observation models that can map high-dimensional tactile images onto a low-dimensional state space. Prior work has used low-dimensional force measurements or hand-designed functions to interpret tactile measurements. These methods, however, can be brittle and difficult to scale across objects and sensors. Our key insight is to directly learn tactile observation models that predict the relative pose of the sensor given a pair of tactile images. These relative poses can then be incorporated as factors within a factor graph. We propose a two-stage approach: first we learn local tactile observation models supervised with ground truth data, and then integrate these models along with physics and geometric factors within a factor graph optimizer. We demonstrate reliable object tracking using only tactile feedback for over 150 real-world planar pushing sequences with varying trajectories across three object shapes. Supplementary video: this https URL
Learning to extrapolate using continued fractions: Predicting the critical temperature of superconductor materials

In the field of Artificial Intelligence (AI) and Machine Learning (ML), a common objective is the approximation of unknown target functions y=f(x) using limited instances S=(x(i),y(i)), where x(i)∈D and D represents the domain of interest. We refer to S as the training set and aim to identify a low-complexity mathematical model that can effectively approximate this target function for new instances x. Consequently, the model’s generalization ability is evaluated on a separate set T={x(j)}⊂D, where T≠S, frequently with T∩S=∅, to assess its performance beyond the training set. However, certain applications require accurate approximation not only within the original domain D but in an extended domain D′ that encompasses D as well. This becomes particularly relevant in scenarios involving the design of new structures, where minimizing errors in approximations is crucial. For example, when developing new materials through data-driven approaches, the AI/ML system can provide valuable insights to guide the design process by serving as a surrogate function. Consequently, the learned model can be employed to facilitate the design of new laboratory experiments. In this paper, we propose a method for multivariate regression based on iterative fitting of a continued fraction, incorporating additive spline models. We compare the performance of our method with established techniques, including AdaBoost, Kernel Ridge, Linear Regression, Lasso Lars, Linear Support Vector Regression, Multi-Layer Perceptrons, Random Forest, Stochastic Gradient Descent, and XGBoost. To evaluate these methods, we focus on an important problem in the field, namely, predicting the critical temperature of superconductors based on their physical–chemical characteristics.
Graph Mixture Density Networks

We introduce the Graph Mixture Density Network, a new family of machine learning models that can fit multimodal output distributions conditioned on arbitrary input graphs. By combining ideas from mixture models and graph representation learning, we address a broad class of challenging regression problems that rely on structured data. Our main contribution is the design and evaluation of our method on large stochastic epidemic simulations conditioned on random graphs. We show that there is a significant improvement in the likelihood of an epidemic outcome when taking into account both multimodality and structure. In addition, we investigate how to \textit{implicitly} retain structural information in node representations by computing the distance between distributions of adjacent nodes, and the technique is tested on two structure reconstruction tasks with very good accuracy. Graph Mixture Density Networks open appealing research opportunities in the study of structure-dependent phenomena that exhibit non-trivial conditional output distributions.
Cooperative Ressource Sharing With Adamant Player

Cooperative game theory deals with systems where players want to cooperate to improve their payoffs. But players may choose coalitions in a non-cooperative manner, leading to a coalition-formation game. We consider such a game with several players (willing to cooperate) and an adamant player (unwilling to cooperate) involved in resource-sharing. Here, the strategy of a player is the set of players with whom it wants to form a coalition. Given a strategy profile, an appropriate partition of coalitions is formed; players in each coalition maximize their collective utilities leading to a non-cooperative resource-sharing game among the coalitions, the utilities at the resulting equilibrium are shared via Shapley-value; these shares define the utilities of players for the given strategy profile in coalition-formation game. We also consider the utilitarian solution to derive the price of anarchy (PoA). We considered a case with symmetric players and an adamant player; wherein we observed that players prefer to stay alone at Nash equilibrium when the number of players (n) is more than 4. In contrast, in the majority of the cases, the utilitarian partition is grand coalition. Interestingly the PoA is smaller with an adamant player of intermediate strength. Further, PoA grows like O(n).
PFA-GAN: Progressive Face Aging With Generative Adversarial Network

Face aging is to render a given face to predict its future appearance, which plays an important role in the information forensics and security field as the appearance of the face typically varies with age. Although impressive results have been achieved with conditional generative adversarial networks (cGANs), the existing cGANs-based methods typically use a single network to learn various aging effects between any two different age groups. However, they cannot simultaneously meet three essential requirements of face aging—including image quality, aging accuracy, and identity preservation—and usually generate aged faces with strong ghost artifacts when the age gap becomes large. Inspired by the fact that faces gradually age over time, this paper proposes a novel progressive face aging framework based on generative adversarial network (PFA-GAN) to mitigate these issues. Unlike the existing cGANs-based methods, the proposed framework contains several sub-networks to mimic the face aging process from young to old, each of which only learns some specific aging effects between two adjacent age groups. The proposed framework can be trained in an end-to-end manner to eliminate accumulative artifacts and blurriness. Moreover, this paper introduces an age estimation loss to take into account the age distribution for an improved aging accuracy, and proposes to use the Pearson correlation coefficient as an evaluation metric measuring the aging smoothness for face aging methods. Extensively experimental results demonstrate superior performance over existing (c)GANs-based methods, including the state-of-the-art one; e.g., PFA-GAN reduces the aging estimation errors by 0.23 and 0.35 and increases the identity preservation rates by 0.49 and 0.63 on two benchmarked datasets compared to the second best method for the challenging face aging from 30− to 51+. The source code is available at https://github.com/Hzzone/PFA-GAN.
Variational Autoencoders for Learning Nonlinear Dynamics of PDEs and Reductions

We develop data-driven methods for incorporating physical information for priors to learn parsimonious representations of nonlinear systems arising from parameterized PDEs and mechanics. Our approach is based on Variational Autoencoders (VAEs) for learning from observations nonlinear state space models. We develop ways to incorporate geometric and topological priors through general manifold latent space representations. We investigate the performance of our methods for learning low dimensional representations for the nonlinear Burgers equation and constrained mechanical systems.
A Modern Primer on Processing in Memory

Modern computing systems are overwhelmingly designed to move data to computation. This design choice goes directly against at least three key trends in computing that cause performance, scalability and energy bottlenecks: (1) data access is a key bottleneck as many important applications are increasingly data-intensive, and memory bandwidth and energy do not scale well, (2) energy consumption is a key limiter in almost all computing platforms, especially server and mobile systems, (3) data movement, especially off-chip to on-chip, is very expensive in terms of bandwidth, energy and latency, much more so than computation. These trends are especially severely-felt in the data-intensive server and energy-constrained mobile systems of today. At the same time, conventional memory technology is facing many technology scaling challenges in terms of reliability, energy, and performance. As a result, memory system architects are open to organizing memory in different ways and making it more intelligent, at the expense of higher cost. The emergence of 3D-stacked memory plus logic, the adoption of error correcting codes inside the latest DRAM chips, proliferation of different main memory standards and chips, specialized for different purposes (e.g., graphics, low-power, high bandwidth, low latency), and the necessity of designing new solutions to serious reliability and security issues, such as the RowHammer phenomenon, are an evidence of this trend. This chapter discusses recent research that aims to practically enable computation close to data, an approach we call processing-in-memory (PIM). PIM places computation mechanisms in or near where the data is stored (i.e., inside the memory chips, in the logic layer of 3D-stacked memory, or in the memory controllers), so that data movement between the computation units and memory is reduced or eliminated.
Knowledge Distillation Thrives on Data Augmentation

Knowledge distillation (KD) is a general deep neural network training framework that uses a teacher model to guide a student model. Many works have explored the rationale for its success, however, its interplay with data augmentation (DA) has not been well recognized so far. In this paper, we are motivated by an interesting observation in classification: KD loss can benefit from extended training iterations while the cross-entropy loss does not. We show this disparity arises because of data augmentation: KD loss can tap into the extra information from different input views brought by DA. By this explanation, we propose to enhance KD via a stronger data augmentation scheme (e.g., mixup, CutMix). Furthermore, an even stronger new DA approach is developed specifically for KD based on the idea of active learning. The findings and merits of the proposed method are validated by extensive experiments on CIFAR-100, Tiny ImageNet, and ImageNet datasets. We can achieve improved performance simply by using the original KD loss combined with stronger augmentation schemes, compared to existing state-of-the-art methods, which employ more advanced distillation losses. In addition, when our approaches are combined with more advanced distillation losses, we can advance the state-of-the-art performance even more. On top of the encouraging performance, this paper also sheds some light on explaining the success of knowledge distillation. The discovered interplay between KD and DA may inspire more advanced KD algorithms.
Amortized Q-learning with Model-based Action Proposals for Autonomous Driving on Highways

Well-established optimization-based methods can guarantee an optimal trajectory for a short optimization horizon, typically no longer than a few seconds. As a result, choosing the optimal trajectory for this short horizon may still result in a sub-optimal long-term solution. At the same time, the resulting short-term trajectories allow for effective, comfortable and provable safe maneuvers in a dynamic traffic environment. In this work, we address the question of how to ensure an optimal long-term driving strategy, while keeping the benefits of classical trajectory planning. We introduce a Reinforcement Learning based approach that coupled with a trajectory planner, learns an optimal long-term decision-making strategy for driving on highways. By online generating locally optimal maneuvers as actions, we balance between the infinite low-level continuous action space, and the limited flexibility of a fixed number of predefined standard lane-change actions. We evaluated our method on realistic scenarios in the open-source traffic simulator SUMO and were able to achieve better performance than the 4 benchmark approaches we compared against, including a random action selecting agent, greedy agent, high-level, discrete actions agent and an IDM-based SUMO-controlled agent.
Computer Stereo Vision for Autonomous Driving

As an important component of autonomous systems, autonomous car perception has had a big leap with recent advances in parallel computing architectures. With the use of tiny but full-feature embedded supercomputers, computer stereo vision has been prevalently applied in autonomous cars for depth perception. The two key aspects of computer stereo vision are speed and accuracy. They are both desirable but conflicting properties, as the algorithms with better disparity accuracy usually have higher computational complexity. Therefore, the main aim of developing a computer stereo vision algorithm for resource-limited hardware is to improve the trade-off between speed and accuracy. In this chapter, we introduce both the hardware and software aspects of computer stereo vision for autonomous car systems. Then, we discuss four autonomous car perception tasks, including 1) visual feature detection, description and matching, 2) 3D information acquisition, 3) object detection/recognition and 4) semantic image segmentation. The principles of computer stereo vision and parallel computing on multi-threading CPU and GPU architectures are then detailed.
Pose-Guided Human Animation from a Single Image in the Wild

We present a new pose transfer method for synthesizing a human animation from a single image of a person controlled by a sequence of body poses. Existing pose transfer methods exhibit significant visual artifacts when applying to a novel scene, resulting in temporal inconsistency and failures in preserving the identity and textures of the person. To address these limitations, we design a compositional neural network that predicts the silhouette, garment labels, and textures. Each modular network is explicitly dedicated to a subtask that can be learned from the synthetic data. At the inference time, we utilize the trained network to produce a unified representation of appearance and its labels in UV coordinates, which remains constant across poses. The unified representation provides an incomplete yet strong guidance to generating the appearance in response to the pose change. We use the trained network to complete the appearance and render it with the background. With these strategies, we are able to synthesize human animations that can preserve the identity and appearance of the person in a temporally coherent way without any fine-tuning of the network on the testing scene. Experiments show that our method outperforms the state-of-the-arts in terms of synthesis quality, temporal coherence, and generalization ability.
Synthesis, X-Ray Crystal Structure and Tubulin- Binding Properties of a Benzofuran Analogue of the Potent Cytotoxic Agent Combretastatin A4

The benzofuran (4), a ring-fused analogue of the potent antimitotic agent combretastatin A4 (1), has been prepared by a convergent route involving 5-endo-dig iodocyclization of o-hydroxytolan (5) as the key step. Compound (4), which has been characterized crystallographically as well as spectroscopically, is inactive as a tubulin-binding agent.
DiffPrune: Neural Network Pruning with Deterministic Approximate Binary Gates and L0 Regularization

Modern neural network architectures typically have many millions of parameters and can be pruned significantly without substantial loss in effectiveness which demonstrates they are over-parameterized. The contribution of this work is two-fold. The first is a method for approximating a multivariate Bernoulli random variable by means of a deterministic and differentiable transformation of any real-valued multivariate random variable. The second is a method for model selection by element-wise multiplication of parameters with approximate binary gates that may be computed deterministically or stochastically and take on exact zero values. Sparsity is encouraged by the inclusion of a surrogate regularization to the $L_0$ loss. Since the method is differentiable it enables straightforward and efficient learning of model architectures by an empirical risk minimization procedure with stochastic gradient descent and theoretically enables conditional computation during training. The method also supports any arbitrary group sparsity over parameters or activations and therefore offers a framework for unstructured or flexible structured model pruning. To conclude experiments are performed to demonstrate the effectiveness of the proposed approach.
Generic Semi-Supervised Adversarial Subject Translation for Sensor-Based Human Activity Recognition

The performance of Human Activity Recognition (HAR) models, particularly deep neural networks, is highly contingent upon the availability of the massive amount of annotated training data which should be sufficiently labeled. Though, data acquisition and manual annotation in the HAR domain are prohibitively expensive due to skilled human resource requirements in both steps. Hence, domain adaptation techniques have been proposed to adapt the knowledge from the existing source of data. More recently, adversarial transfer learning methods have shown very promising results in image classification, yet limited for sensor-based HAR problems, which are still prone to the unfavorable effects of the imbalanced distribution of samples. This paper presents a novel generic and robust approach for semi-supervised domain adaptation in HAR, which capitalizes on the advantages of the adversarial framework to tackle the shortcomings, by leveraging knowledge from annotated samples exclusively from the source subject and unlabeled ones of the target subject. Extensive subject translation experiments are conducted on three large, middle, and small-size datasets with different levels of imbalance to assess the robustness and effectiveness of the proposed model to the scale as well as imbalance in the data. The results demonstrate the effectiveness of our proposed algorithms over state-of-the-art methods, which led in up to 13%, 4%, and 13% improvement of our high-level activities recognition metrics for Opportunity, LISSI, and PAMAP2 datasets, respectively. The LISSI dataset is the most challenging one owing to its less populated and imbalanced distribution. Compared to the SA-GAN adversarial domain adaptation method, the proposed approach enhances the final classification performance with an average of 7.5% for the three datasets, which emphasizes the effectiveness of micro-mini-batch training.
SLITRONOMY: Towards a fully wavelet-based strong lensing inversion technique

Strong gravitational lensing provides a wealth of astrophysical information on the baryonic and dark matter content of galaxies. It also serves as a valuable cosmological probe by allowing us to measure the Hubble constant independently of other methods. These applications all require the difficult task of inverting the lens equation and simultaneously reconstructing the mass profile of the lens along with the original light profile of the unlensed source. As there is no reason for either the lens or the source to be simple, we need methods that both invert the lens equation with a large number of degrees of freedom and also enforce a well-controlled regularisation that avoids the appearance of spurious structures. This can be beautifully accomplished by representing signals in wavelet space. Building on the Sparse Lens Inversion Technique (SLIT), we present an improved sparsity-based method that describes lensed sources using wavelets and optimises over the parameters given an analytical lens mass profile. We applied our technique on simulated HST and E-ELT data, as well as on real HST images of lenses from the Sloan Lens ACS sample, assuming a lens model. We show that wavelets allowed us to reconstruct lensed sources containing detailed substructures when using both present-day data and very high-resolution images expected from future thirty-metre-class telescopes. In the latter case, wavelets moreover provide a much more tractable solution in terms of quality and computation time compared to using a source model that combines smooth analytical profiles and shapelets. Requiring very little human interaction, our flexible pixel-based technique fits into the ongoing effort to devise automated modelling schemes. It can be incorporated in the standard workflow of sampling analytical lens model parameters while modelling the source on a pixelated grid. The method, which we call SLITRONOMY, is freely available as a new plug-in to the modelling software LENSTRONOMY.
Stochastic SPOD-Galerkin model of a turbulent jet.

The use of spectral proper orthogonal decomposition (SPOD) to construct low-order models for broadband turbulent flows is explored. The choice of SPOD modes as basis vectors is motivated by their optimality and space-time coherence properties for statistically stationary flows. This work follows the modeling paradigm that complex nonlinear fluid dynamics can be approximated as stochastically forced linear systems. The proposed model combines ideas from Galerkin projection and linear multi-level regression (MLR) modeling. The resulting stochastic two-level SPOD-Galerkin model governs a compound state consisting of the modal expansion coefficients and forcing coefficients. In the first level, the modal expansion coefficients are advanced by the forced linearized Navier-Stokes operator under the linear time-invariant assumption. The second level governs the forcing coefficients, which compensate for the offset between the linear approximation and the true state. At this level, least squares regression is used to achieve closure by modeling nonlinear interactions between modes. The statistics of the remaining residue are used to construct a dewhitening filter. This filter facilitates the use of white noise to drive the stochastic flow model. The model is demonstrated on large eddy simulation data of a turbulent jet at Mach number 0.9 and a Reynolds number of approximately 1M. The surrogate data obtained from the model accurately reproduces the second-order statistics and dynamics of the input data, and instantaneous instances of the surrogate flow field are visually indistinguishable from the original data. The model uncertainty, predictability, and stability are quantified analytically, when possible, and through Monte Carlo simulation.