Miracle-dz/UniFAR-FTUs huggingface dataset:
Continuous vs. Discrete Optimization of Deep Neural Networks

Existing analyses of optimization in deep learning are either continuous, focusing on (variants of) gradient flow, or discrete, directly treating (variants of) gradient descent. Gradient flow is amenable to theoretical analysis, but is stylized and disregards computational efficiency. The extent to which it represents gradient descent is an open question in the theory of deep learning. The current paper studies this question. Viewing gradient descent as an approximate numerical solution to the initial value problem of gradient flow, we find that the degree of approximation depends on the curvature around the gradient flow trajectory. We then show that over deep neural networks with homogeneous activations, gradient flow trajectories enjoy favorable curvature, suggesting they are well approximated by gradient descent. This finding allows us to translate an analysis of gradient flow over deep linear neural networks into a guarantee that gradient descent efficiently converges to global minimum almost surely under random initialization. Experiments suggest that over simple deep neural networks, gradient descent with conventional step size is indeed close to gradient flow. We hypothesize that the theory of gradient flows will unravel mysteries behind deep learning.
Stochastic Geometry based Interference Analysis of Multiuser mmWave Networks with RIS

In this paper, we utilize tools from stochastic geometry to estimate the interference propagation via reconfigurable intelligent surface (RIS) in the millimeter wave (mmWave, 30–300 GHz) band and specifically on the D band (110–170 GHz). The RISs have been of great interest lately to maximize the channel gains in non-line-of-sight (NLOS) communication situations. We derive expressions for stochastic interference level in RIS powered systems and validate those with simulations. It will be shown that the interference levels via RIS link are rather small compared to the designed RIS link or the LOS interference as the random interference loses significant part of the RIS gain. We also analyse the validity of far field channel and antenna gains in the near field of a large array. It is shown that, while the high frequency systems require large arrays that push the far field far away from the antenna, the far field equations are very accurate up to about half way of the near field.
Does Parking Matter? The Impact of Search Time for Parking on Last-Mile Delivery Optimization

Parking is a necessary component of traditional last-mile delivery practices, but finding parking can be difficult. Yet, the routing literature largely does not account for the need to find parking. In this paper, we address this challenge of finding parking through the Capacitated Delivery Problem with Parking (CDPP). Unlike other models in the literature, the CDPP accounts for the search time for parking in the objective and minimizes the completion time of the delivery tour. When we restrict the customer geography to a complete grid, we identify conditions for when a Traveling Salesman Problem (TSP) solution that parks at each customer is an optimal solution to the CDPP. We then determine when the search time for parking is large enough for the CDPP optimal solution to differ from this TSP solution. We also identify model improvements that allow reasonably-sized instances of the CDPP to be solved exactly. We introduce a heuristic for the CDPP that quickly finds high quality solutions to large instances. Computational experiments show that parking matters in last-mile delivery optimization. The CDPP outperforms industry practice and models in the literature showing the greatest advantage when the search time for parking is high. This analysis provides immediate ways to improve routing in last-mile delivery. 1 ar X iv :2 10 7. 06 78 8v 2 [ m at h. O C ] 3 1 Ja n 20 22
MMGCN: Multimodal Fusion via Deep Graph Convolution Network for Emotion Recognition in Conversation

Emotion recognition in conversation (ERC) is a crucial component in affective dialogue systems, which helps the system understand users’ emotions and generate empathetic responses. However, most works focus on modeling speaker and contextual information primarily on the textual modality or simply leveraging multimodal information through feature concatenation. In order to explore a more effective way of utilizing both multimodal and long-distance contextual information, we propose a new model based on multimodal fused graph convolutional network, MMGCN, in this work. MMGCN can not only make use of multimodal dependencies effectively, but also leverage speaker information to model inter-speaker and intra-speaker dependency. We evaluate our proposed model on two public benchmark datasets, IEMOCAP and MELD, and the results prove the effectiveness of MMGCN, which outperforms other SOTA methods by a significant margin under the multimodal conversation setting.
Exploring the Potential of Low-Bit Training of Convolutional Neural Networks

Convolutional neural networks (CNNs) have been widely used in many tasks, but training CNNs is time consuming and energy hungry. Using the low-bit integer format has been proved promising for speeding up and improving the energy efficiency of CNN inference, while CNN training can hardly benefit from such a technique because of the following challenges: 1) the integer data format cannot meet the requirements of the data dynamic range in training, resulting in the accuracy drop; 2) the floating-point data format keeps sizeable dynamic range with much more exponent bits, thus using it results in higher accumulation power than using the integer data format; and 3) there are some specially designed data formats (e.g., with group-wise scaling) that have the potential to deal with the former two problems but common hardware platforms cannot support them efficiently. To tackle all these challenges and make the training phase of CNNs benefit from the low-bit format, we propose a low-bit training framework for CNNs to pursue a better tradeoff between accuracy and energy efficiency: 1) we adopt element-wise scaling to increase the dynamic range of data representation, which significantly reduces the quantization error; 2) group-wise scaling with hardware friendly factor format is designed to reduce the element-wise exponent bits without degrading the accuracy; and 3) we design the customized hardware unit that implements the low-bit tensor convolution arithmetic with our multilevel scaling data format. Experiments show that our framework achieves a superior tradeoff between the accuracy and the bit-width than previous low-bit training studies. For training various models on CIFAR-10, using 1-bit mantissa and 2-bit exponent is adequate to keep the accuracy loss within 1%. On larger datasets like ImageNet, using 4-bit mantissa and 2-bit exponent is adequate. Through the energy consumption simulation of the whole network, we can see that training a variety of models with our framework could achieve $4.9\times $ – $10.2\times $ higher energy efficiency than full-precision arithmetic.
Parametrization of Nonbonded Force Field Terms for Metal-Organic Frameworks Using Machine Learning Approach

The enormous structural and chemical diversity of metal-organic frameworks (MOFs) forces researchers to actively use simulation techniques as often as experiments. MOFs are widely known for their outstanding adsorption properties, so a precise description of the host-guest interactions is essential for high-throughput screening aimed at ranking the most promising candidates. However, highly accurate ab initio calculations cannot be routinely applied to model thousands of structures due to the demanding computational costs. Furthermore, methods based on force field (FF) parametrization suffer from low transferability. To resolve this accuracy-efficiency dilemma, we applied a machine learning (ML) approach: extreme gradient boosting. The trained models reproduced the atom-in-material quantities, including partial charges, polarizabilities, dispersion coefficients, quantum Drude oscillator, and electron cloud parameters, with accuracy similar to the reference data set. The aforementioned FF precursors make it possible to thoroughly describe noncovalent interactions typical for MOF-adsorbate systems: electrostatic, dispersion, polarization, and short-range repulsion. The presented approach can also readily facilitate hybrid atomistic simulation/ML workflows.
Precise Visual-Inertial Localization for UAV with the Aid of A 2D Georeferenced Map

— Precise geolocalization is crucial for unmanned aerial vehicles (UAVs). However, most current deployed UAVs rely on the global navigation satellite systems (GNSS) or high precision inertial navigation systems (INS) for geolocalization. In this paper, we propose to use a lightweight visual-inertial system with a 2D georeference map to obtain accurate and consecutive geodetic positions for UAVs. The proposed system firstly integrates a micro inertial measurement unit (MIMU) and a monocular camera as odometry to consecutively estimate the navigation states and reconstruct the 3D position of the observed visual features in the local world frame. To obtain the geolocation, the visual features tracked by the odometry are further registered to the 2D georeferenced map. While most conventional methods perform image-level aerial image registration, we propose to align the reconstructed points to the map points in the geodetic frame; this helps to filter out the large portion of outliers and decouples the negative effects from the horizontal angles. The registered points are then used to relocalize the vehicle in the geodetic frame. Finally, a pose graph is deployed to fuse the geolocation from the aerial image registration and the local navigation result from the visual-inertial odometry (VIO) to achieve consecutive and drift-free geolocalization performance. We have validated the proposed method by installing the sensors to a UAV body rigidly and have conducted two flights in different environments with unknown initials. The results show that the proposed method can achieve less than 4m position error in flight at 100m high and less than 9m position error in flight about 300m high.
Cats, not CAT scans: a study of dataset similarity in transfer learning for 2D medical image classification

Transfer learning is a commonly used strategy for medical image classification, especially via pretraining on source data and fine-tuning on target data. There is currently no consensus on how to choose appropriate source data, and in the literature we can find both evidence of favoring large natural image datasets such as ImageNet, and evidence of favoring more specialized medical datasets. In this paper we perform a systematic study with nine source datasets with natural or medical images, and three target medical datasets, all with 2D images. We find that ImageNet is the source leading to the highest performances, but also that larger datasets are not necessarily better. We also study different definitions of data similarity. We show that common intuitions about similarity may be inaccurate, and therefore not sufficient to predict an appropriate source a priori. Finally, we discuss several steps needed for further research in this field, especially with regard to other types (for example 3D) medical images. Our experiments and pretrained models are available via \url{https://www.github. com/vcheplygina/cats-scans}
The Climate in Climate Economics

To analyze climate change mitigation strategies, economists rely on simplified climate models – so-called climate emulators – that provide a realistic quantitative link between CO2 emissions and global warming at low computational costs. In this paper, we propose a generic and transparent calibration and evaluation strategy for these climate emulators that is based on freely and easily accessible state-of-the-art benchmark data from climate sciences. We demonstrate that the appropriate choice of the free model parameters can be of key relevance for the predicted social cost of carbon. The key idea we put forward is to calibrate the simplified climate models to benchmark data from comprehensive global climate models that took part in the Coupled Model Intercomparison Project, Phase 5 (CMIP5). In particular, we propose to use four different test cases that are considered pivotal in the climate science literature: two highly idealized tests to separately calibrate and evaluate the carbon cycle and temperature response, an idealized test to quantify the transient climate response, and a final test to evaluate the performance for scenarios close to those arising from economic models, and that include exogenous forcing. As a concrete example, we re-calibrate the climate part of the widely used DICE-2016, fathoming the CMIP5 uncertainty range of model responses: the multi-model mean as well as extreme, but still permissible climate sensitivities and carbon cycle responses. We demonstrate that the functional form of the climate emulator of the DICE-2016 model is fit for purpose, despite its simplicity, but its carbon cycle and temperature equations are miscalibrated, leading to the conclusion that one may want to be skeptical about predictions derived from DICE-2016. We examine the importance of the calibration for the social cost of carbon in the context of a partial equilibrium setting where interest rates are exogenous, as well as the simple general equilibrium setting from DICE-2016. We find that the model uncertainty from different consistent calibrations of the climate system can change the social cost of carbon by a factor of four if one assumes a quadratic damage function. When calibrated to the multi-model mean, our model predicts similar values for the social cost of carbon as the original DICE-2016, but with a strongly reduced sensitivity to the discount rate and about one degree less long-term warming. The social cost of carbon in DICE-2016 is oversensitive to the discount rate, leading to extreme comparative statics responses to changes in preferences.
Hierarchical Associative Memory

Dense Associative Memories or Modern Hopfield Networks have many appealing properties of associative memory. They can do pattern completion, store a large number of memories, and can be described using a recurrent neural network with a degree of biological plausibility and rich feedback between the neurons. At the same time, up until now all the models of this class have had only one hidden layer, and have only been formulated with densely connected network architectures, two aspects that hinder their machine learning applications. This paper tackles this gap and describes a fully recurrent model of associative memory with an arbitrary large number of layers, some of which can be locally connected (convolutional), and a corresponding energy function that decreases on the dynamical trajectory of the neurons' activations. The memories of the full network are dynamically"assembled"using primitives encoded in the synaptic weights of the lower layers, with the"assembling rules"encoded in the synaptic weights of the higher layers. In addition to the bottom-up propagation of information, typical of commonly used feedforward neural networks, the model described has rich top-down feedback from higher layers that help the lower-layer neurons to decide on their response to the input stimuli.
On the Spin Dynamics of Elongated Minor Bodies with Applications to a Possible Solar System Analogue Composition for ‘Oumuamua

The first interstellar object, 1I/2017 U1 (‘Oumuamua), exhibited several unique properties, including an extreme aspect ratio, a lack of typical cometary volatiles, and a deviation from a Keplerian trajectory. Several authors have hypothesized that the non-gravitational acceleration was caused by either cometary outgassing or radiation pressure. Here, we investigate the spin dynamics of ‘Oumuamua under the action of high-surface-area fractional activity and radiation pressure. We demonstrate that a series of transient jets that migrate across the illuminated surface will not produce a secular increase in the spin rate. We produce 3D tumbling simulations that approximate the dynamics of a surface-covering jet and show that the resulting synthetic light curve and periodogram are reasonably consistent with the observations. Moreover, we demonstrate that radiation pressure also produces a steady spin state. While carbon monoxide (CO) has been dismissed as a possible accelerant because of its non-detection in emission by Spitzer, we show that outgassing from a surface characterized by a modest covering fraction of CO ice can satisfy the non-ballistic dynamics for a plausible range of assumed bulk densities and surface albedos. Spitzer upper limits on CO emission are, however, inconsistent with the CO production necessary to provide the acceleration. Nonetheless, an ad hoc but physically plausible explanation is that the activity level varied greatly during the time that the trajectory was monitored. We reproduce the astrometric analysis presented in Micheli et al., and verify that the non-gravitational acceleration was consistent with stochastic changes in outgassing.
Per-Pixel Classification is Not All You Need for Semantic Segmentation

Modern approaches typically formulate semantic segmentation as a per-pixel classification task, while instance-level segmentation is handled with an alternative mask classification. Our key insight: mask classification is sufficiently general to solve both semantic- and instance-level segmentation tasks in a unified manner using the exact same model, loss, and training procedure. Following this observation, we propose MaskFormer, a simple mask classification model which predicts a set of binary masks, each associated with a single global class label prediction. Overall, the proposed mask classification-based method simplifies the landscape of effective approaches to semantic and panoptic segmentation tasks and shows excellent empirical results. In particular, we observe that MaskFormer outperforms per-pixel classification baselines when the number of classes is large. Our mask classification-based method outperforms both current state-of-the-art semantic (55.6 mIoU on ADE20K) and panoptic segmentation (52.7 PQ on COCO) models.
Optimal Input Representation in Neural Systems at the Edge of Chaos

Simple Summary Here we show that a simple neural network within the paradigm of reservoir computing is able to reproduce an important feature of internal representations of neural inputs, in agreement with what theoretically predicted and empirically measured in the mouse visual cortex, only when it is set to operate at the edge of chaos. Abstract Shedding light on how biological systems represent, process and store information in noisy environments is a key and challenging goal. A stimulating, though controversial, hypothesis poses that operating in dynamical regimes near the edge of a phase transition, i.e., at criticality or the “edge of chaos”, can provide information-processing living systems with important operational advantages, creating, e.g., an optimal trade-off between robustness and flexibility. Here, we elaborate on a recent theoretical result, which establishes that the spectrum of covariance matrices of neural networks representing complex inputs in a robust way needs to decay as a power-law of the rank, with an exponent close to unity, a result that has been indeed experimentally verified in neurons of the mouse visual cortex. Aimed at understanding and mimicking these results, we construct an artificial neural network and train it to classify images. We find that the best performance in such a task is obtained when the network operates near the critical point, at which the eigenspectrum of the covariance matrix follows the very same statistics as actual neurons do. Thus, we conclude that operating near criticality can also have—besides the usually alleged virtues—the advantage of allowing for flexible, robust and efficient input representations.
Age of Information in Physical-Layer Network Coding Enabled Two-Way Relay Networks

This paper investigates the information freshness of two-way relay networks (TWRNs) operated with physical-layer network coding (PNC). Information freshness is quantified by age of information (AoI), defined as the time elapsed since the generation time of the latest received information update. PNC reduces the communication latency of TWRNs by turning superimposed electromagnetic waves into network-coded messages so that end users can send update packets to each other more frequently via the relay. While sending update packets more frequently has the potential to reduce AoI, how to handle packet corruption in TWRNs has not been investigated. Specifically, if an old packet is corrupted in any hop of a TWRN, one needs to decide whether to drop or to retransmit the old packet, e.g., a new packet has more recent information but may take more time to be delivered. Therefore, we study the average AoI with and without automatic repeat request (ARQ) in PNC-enabled TWRNs. Interestingly, our analysis shows that neither the non-ARQ scheme nor the pure ARQ scheme achieves a good average AoI. Hence, we put forth an uplink-lost-then-drop (ULTD) protocol that combines packet drop and ARQ. Experiments on software-defined radios indicate that ULTD significantly outperforms non-ARQ and pure ARQ schemes in terms of average AoI, especially when the two end users have imbalanced channel conditions. We believe the insight of ULTD on TWRNs generally applies to other two-hop networks: to achieve high information freshness, when packets are corrupted in the first hop, new packets should be generated and sent (i.e., old packets are discarded); when packets are corrupted in the second hop, old packets should be retransmitted until they are successfully received.
The Role of Complex Ionized Absorbers in the Soft X-Ray Spectra of Intermediate Polars

In magnetic cataclysmic variables (mCVs), X-ray radiation originates from a shock-heated, multi-temperature plasma in the post-shock region near the white dwarf’s surface. These X-rays are modified by a complex distribution of absorbers in the pre-shock region. The presence of photoionized lines and warm absorber features in the soft X-ray spectra of these mCVs suggests that these absorbers are ionized. We developed the ionized complex absorber model zxipab, which is represented by a power-law distribution of ionized absorbers in the pre-shock flow. Using the ionized absorber model zxipab along with a cooling flow model with a reflection component, we model the broadband Chandra/High-Energy Transmission Grating (HETG) and Nuclear Spectroscopic Telescope Array (NuSTAR) spectra of two intermediate polars (IPs): NY Lup and V1223 Sgr. We find that this model describes well many of the H- and He-like emission lines from medium-Z elements, which arise from a collisionally excited plasma. However the model fails to account for some of the He-like triplets from the medium-Z elements, which points toward their photoionized origin. We do not find compelling evidence for a blackbody component to model the soft excess seen in the residuals of the Chandra/HETG spectra, which could be due to uncertainties in the estimation of the interstellar absorption of these sources using the Chandra/HETG data and/or excess fluxes seen in some photoionized emission lines which are not accounted for by the cooling flow model. We describe the implications of this model with respect to the geometry of the pre-shock region for these two IPs.