Miracle-dz/UniFAR-FTUs huggingface dataset:
ECRM: Efficient Fault Tolerance for Recommendation Model Training via Erasure Coding

Deep-learning-based recommendation models (DLRMs) are widely deployed to serve personalized content to users. DLRMs are large in size due to their use of large embedding tables, and are trained by distributing the model across the memory of tens or hundreds of servers. Server failures are common in such large distributed systems and must be mitigated to enable training to progress. Checkpointing is the primary approach used for fault tolerance in these systems, but incurs significant training-time overhead both during normal operation and when recovering from failures. As these overheads increase with DLRM size, checkpointing is slated to become an even larger overhead for future DLRMs, which are expected to grow in size. This calls for rethinking fault tolerance in DLRM training. We present ECRM, a DLRM training system that achieves efficient fault tolerance using erasure coding. ECRM chooses which DLRM parameters to encode, correctly and efficiently updates parities, and enables training to proceed without any pauses, while maintaining consistency of the recovered parameters. We implement ECRM atop XDL, an open-source, industrial-scale DLRM training system. Compared to checkpointing, ECRM reduces training-time overhead for large DLRMs by up to 88%, recovers from failures up to 10.3$\times$ faster, and allows training to proceed during recovery. These results show the promise of erasure coding in imparting efficient fault tolerance to training current and future DLRMs.
Hierarchical Pyramid Representations for Semantic Segmentation

Understanding the context of complex and cluttered scenes is a challenging problem for semantic segmentation. However, it is difficult to model the context without prior and additional supervision because the scene's factors, such as the scale, shape, and appearance of objects, vary considerably in these scenes. To solve this, we propose to learn the structures of objects and the hierarchy among objects because context is based on these intrinsic properties. In this study, we design novel hierarchical, contextual, and multiscale pyramidal representations to capture the properties from an input image. Our key idea is the recursive segmentation in different hierarchical regions based on a predefined number of regions and the aggregation of the context in these regions. The aggregated contexts are used to predict the contextual relationship between the regions and partition the regions in the following hierarchical level. Finally, by constructing the pyramid representations from the recursively aggregated context, multiscale and hierarchical properties are attained. In the experiments, we confirmed that our proposed method achieves state-of-the-art performance in PASCAL Context.
Aggregated Contextual Transformations for High-Resolution Image Inpainting

Image inpainting that completes large free-form missing regions in images is a promising yet challenging task. State-of-the-art approaches have achieved significant progress by taking advantage of generative adversarial networks (GAN). However, these approaches can suffer from generating distorted structures and blurry textures in high-resolution images (e.g., <inline-formula><tex-math notation="LaTeX">$512\times 512$</tex-math><alternatives><mml:math><mml:mrow><mml:mn>512</mml:mn><mml:mo>×</mml:mo><mml:mn>512</mml:mn></mml:mrow></mml:math><inline-graphic xlink:href="fu-ieq1-3156949.gif"/></alternatives></inline-formula>). The challenges mainly drive from (1) image content reasoning from distant contexts, and (2) fine-grained texture synthesis for a large missing region. To overcome these two challenges, we propose an enhanced GAN-based model, named <bold>A</bold>ggregated C<bold>O</bold>ntextual-<bold>T</bold>ransformation GAN (<bold>AOT-GAN</bold>), for high-resolution image inpainting. Specifically, to enhance context reasoning, we construct the generator of AOT-GAN by stacking multiple layers of a proposed AOT block. The AOT blocks aggregate contextual transformations from various receptive fields, allowing to capture both informative distant image contexts and rich patterns of interest for context reasoning. For improving texture synthesis, we enhance the discriminator of AOT-GAN by training it with a tailored mask-prediction task. Such a training objective forces the discriminator to distinguish the detailed appearances of real and synthesized patches, and in turn facilitates the generator to synthesize clear textures. Extensive comparisons on Places2, the most challenging benchmark with 1.8 million high-resolution images of 365 complex scenes, show that our model outperforms the state-of-the-art. A user study including more than <bold>30 subjects</bold> further validates the superiority of AOT-GAN. We further evaluate the proposed AOT-GAN in practical applications, e.g., logo removal, face editing, and object removal. Results show that our model achieves promising completions in the real world. We release codes and models in <uri>https://github.com/researchmm/AOT-GAN-for-Inpainting</uri>.
Mitigating Gradient-based Adversarial Attacks via Denoising and Compression

Gradient-based adversarial attacks on deep neural networks pose a serious threat, since they can be deployed by adding imperceptible perturbations to the test data of any network, and the risk they introduce cannot be assessed through the network's original training performance. Denoising and dimensionality reduction are two distinct methods that have been independently investigated to combat such attacks. While denoising offers the ability to tailor the defense to the specific nature of the attack, dimensionality reduction offers the advantage of potentially removing previously unseen perturbations, along with reducing the training time of the network being defended. We propose strategies to combine the advantages of these two defense mechanisms. First, we propose the cascaded defense, which involves denoising followed by dimensionality reduction. To reduce the training time of the defense for a small trade-off in performance, we propose the hidden layer defense, which involves feeding the output of the encoder of a denoising autoencoder into the network. Further, we discuss how adaptive attacks against these defenses could become significantly weak when an alternative defense is used, or when no defense is used. In this light, we propose a new metric to evaluate a defense which measures the sensitivity of the adaptive attack to modifications in the defense. Finally, we present a guideline for building an ordered repertoire of defenses, a.k. a. a defense infrastructure, that adjusts to limited computational resources in presence of uncertainty about the attack strategy.
EKO: Adaptive Sampling of Compressed Video Data

Researchers have presented systems for efficiently analysing video data at scale using sampling algorithms. While these systems effectively leverage the temporal redundancy present in videos, they suffer from three limitations. First, they use traditional video storage formats are tailored for human consumption. Second, they load and decode the entire compressed video in memory before applying the sampling algorithm. Third, the sampling algorithms often require labeled training data obtained using a specific deep learning model. These limitations lead to lower accuracy, higher query execution time, and larger memory footprint. In this paper, we present EKO, a storage engine for efficiently managing video data. EKO relies on two optimizations. First, it uses a novel unsupervised, adaptive sampling algorithm for identifying the key frames in a given video. Second, it stores the identified key frames in a compressed representation that is optimized for machine consumption. We show that EKO improves F1-score by up to 9% compared to the next best performing state-of-the-art unsupervised, sampling algorithms by selecting more representative frames. It reduces query execution time by 3X and memory footprint by 10X in comparison to a widely-used, traditional video storage format.
Using ATOFMS to Determine OC/EC Mass Fractions in Particles

Historically, obtaining quantitative chemical information using laser desorption ionization mass spectrometry for analyzing individual aerosol particles has been quite challenging. This is due in large part to fluctuations in the absolute ion signals resulting from inhomogeneities in the laser beam profile, as well as chemical matrix effects. Progress has been made in quantifying atomic species using high laser powers, but very few studies have been performed quantifying molecular species. In this study, promising results are obtained using a new approach to measure the fraction of organic carbon (OC) associated with elemental carbon (EC) in aerosol particles using single particle laser desorption ionization. A tandem differential mobility analyzer (TDMA) is used to generate OC/EC particles by size selecting EC particles of a given mobility diameter and then coating them with known thicknesses of OC measured using a second DMA. The mass spectra of the OC/EC particles exiting the second DMA are measured using an ultrafine aerosol time-of-flight mass spectrometer (UF-ATOFMS). A calibration curve is produced with a linear correlation (R2 = 0.98) over the range of OC/EC ion intensity ratios observed in source and ambient studies. Importantly, the OC/EC values measured in ambient field tests with the UF-ATOFMS show a linear correlation (R2 = 0.69) with OC/EC mass ratios obtained using semi-continuous filter based thermo-optical measurements. The calibration procedure established herein represents a significant step toward quantification of OC and EC in sub-micron ambient particles using laser desorption ionization mass spectrometry.
Wavelet Design with Optimally Localized Ambiguity Function: a Variational Approach

In this paper, we design mother wavelets for the 1D continuous wavelet transform with some optimality properties. An optimal mother wavelet here is one that has an ambiguity function with minimal spread in the continuous coefficient space (also called phase space). Since the ambiguity function is the reproducing kernel of the coefficient space, optimal windows lead to phase space representations which are"optimally sharp. "Namely, the wavelet coefficients have minimal correlations with each other. Such a construction also promotes sparsity in phase space. The spread of the ambiguity function is modeled as the sum of variances along the axes in phase space. In order to optimize the mother wavelet directly as a 1D signal, we pull-back the variances, defined on the 2D phase space, to the so called window-signal space. This is done using the recently developed wavelet-Plancharel theory. The approach allows formulating the optimization problem of the 2D ambiguity function as a minimization problem of the 1D mother wavelet. The resulting 1D formulation is more efficient and does not involve complicated constraints on the 2D ambiguity function. We optimize the mother wavelet using gradient descent, which yields a locally optimal mother wavelet.
All-dielectric thermonanophotonics

Nanophotonics is an important branch of modern optics dealing with lightmatter interaction at the nanoscale. Nanoparticles can exhibit enhanced light absorption under illumination by light, and they become nanoscale sources of heat that can be precisely controlled and manipulated. For metal nanoparticles, such effects have been studied in the framework of thermoplasmonics which, similar to plasmonics itself, has a number of limitations. Recently emerged all-dielectric resonant nanophotonics is associated with optically-induced electric and magnetic Mie resonances, and this field is developing very rapidly in the last decade. As a result, thermoplasmonics is being replaced by all-dielectric thermonanophotonics with many important applications such as photothermal cancer therapy, drug and gene delivery, nanochemistry, and photothermal imaging. This review paper aims to introduce this new field of non-plasmonic nanophotonics and discuss associated thermally-induced processes at the nanoscale. 1 ar X iv :2 10 4. 01 96 4v 1 [ ph ys ic s. op tic s] 5 A pr 2 02 1
Fast Design Space Exploration of Nonlinear Systems: Part I

System design tools are often only available as input–output blackboxes: for a given design as input, they compute an output representing system behavior. Blackboxes are intended to be run in the forward direction. This article presents a new method of solving the “inverse design problem,” namely, given requirements or constraints on output, find an input that also optimizes an objective function. This problem is challenging for several reasons. First, blackboxes are not designed to be run in reverse. Second, inputs and outputs can be discrete and continuous. Third, finding designs concurrently satisfying a set of requirements is hard because designs satisfying individual requirements may conflict with each other. Fourth, blackbox evaluations can be expensive. Finally, evaluations can sometimes fail to produce an output due to nonconvergence of underlying numerical algorithms. This article presents CNMA, a new method of solving the inverse problem that overcomes these challenges. CNMA tries to sample only the part of the design space relevant to solving the inverse problem, leveraging the power of neural networks, mixed-integer linear programs, and a new learning-from-failure feedback loop. This article also presents a parallel version of CNMA that improves the efficiency and quality of solutions over the sequential version and tries to steer it away from local optima. CNMA’s performance is evaluated against conventional optimization methods for seven nonlinear design problems of 8 (two problems), 10, 15, 36 and 60 real-valued dimensions and one with 186 binary dimensions. Conventional methods evaluated are stable, off-the-shelf implementations of the Bayesian optimization with the Gaussian Processes, Nelder–Mead, and Random Search. The first two do not even produce a solution for problems that are high dimensional, having both discrete and continuous variables or whose blackboxes fail to return values for some inputs. CNMA produces solutions for all problems. When conventional methods do produce solutions, CNMA improves upon their performance by 1%–87%.
Tight Compression: Compressing CNN Through Fine-Grained Pruning and Weight Permutation for Efficient Implementation

The unstructured sparsity after pruning poses a challenge to the efficient implementation of deep learning models in existing regular architectures like systolic arrays. On the other hand, coarse-grained structured pruning is suitable for implementation in regular architectures but tends to have higher accuracy loss than unstructured pruning when the pruned models are of the same size. In this work, we propose a model compression method based on a novel weight permutation scheme to fully exploit the fine-grained weight sparsity in the hardware design. Through permutation, the optimal arrangement of the weight matrix is obtained, and the sparse weight matrix is further compressed to a small and dense format to make full use of the hardware resources. Two pruning granularities are explored. In addition to the unstructured weight pruning, we also propose a more fine-grained subword-level pruning to further improve the compression performance. Compared to the state-of-the-art works, the matrix compression rate is significantly improved from <inline-formula> <tex-math notation="LaTeX">$5.88\times $ </tex-math></inline-formula> to <inline-formula> <tex-math notation="LaTeX">$14.13\times $ </tex-math></inline-formula>. As a result, the throughput and energy efficiency are improved by 2.75 and 1.86 times, respectively.
Regulation of the normalized rate of driven magnetic reconnection through shocked flux pileup

Magnetic reconnection is explored on the Terrestrial Reconnection Experiment (TREX) for asymmetric inflow conditions and in a configuration where the absolute rate of reconnection is set by an external drive. Magnetic pileup enhances the upstream magnetic field of the high-density inflow, leading to an increased upstream Alfvén speed and helping to lower the normalized reconnection rate to values expected from theoretical consideration. In addition, a shock interface between the far upstream supersonic plasma inflow and the region of magnetic flux pileup is observed, important to the overall force balance of the system, thereby demonstrating the role of shock formation for configurations including a supersonically driven inflow. Despite the specialized geometry where a strong reconnection drive is applied from only one side of the reconnection layer, previous numerical and theoretical results remain robust and are shown to accurately predict the normalized rate of reconnection for the range of system sizes considered. This experimental rate of reconnection is dependent on system size, reaching values as high as 0.8 at the smallest normalized system size applied.
UDO: Universal Database Optimization using Reinforcement Learning

UDO is a versatile tool for offline tuning of database systems for specific workloads. UDO can consider a variety of tuning choices, reaching from picking transaction code variants over index selections up to database system parameter tuning. UDO uses reinforcement learning to converge to near-optimal configurations, creating and evaluating different configurations via actual query executions (instead of relying on simplifying cost models). To cater to different parameter types, UDO distinguishes heavy parameters (which are expensive to change, e.g. physical design parameters) from light parameters. Specifically for optimizing heavy parameters, UDO uses reinforcement learning algorithms that allow delaying the point at which the reward feedback becomes available. This gives us the freedom to optimize the point in time and the order in which different configurations are created and evaluated (by benchmarking a workload sample). UDO uses a cost-based planner to minimize reconfiguration overheads. For instance, it aims to amortize the creation of expensive data structures by consecutively evaluating configurations using them. We evaluate UDO on Postgres as well as MySQL and on TPC-H as well as TPC-C, optimizing a variety of light and heavy parameters concurrently.
Safe Control Synthesis via Input Constrained Control Barrier Functions

This paper introduces the notion of an Input Constrained Control Barrier Function (ICCBF), as a method to synthesize safety-critical controllers for nonlinear control-affine systems with input constraints. The method identifies a subset of the safe set of states, and constructs a controller to render the subset forward invariant. The feedback controller is represented as the solution to a quadratic program, which can be solved efficiently for real-time implementation. Furthermore, we show that ICCBFs are a generalization of Higher Order Control Barrier Functions, and thus are applicable to systems of nonuniform relative degree. Simulation results are presented for the adaptive cruise control problem, and a spacecraft rendezvous problem.
TransfoRNN: Capturing the Sequential Information in Self-Attention Representations for Language Modeling

In this paper, we describe the use of recurrent neural networks to capture sequential information from the self-attention representations to improve the Transformers. Although self-attention mechanism provides a means to exploit long context, the sequential information, i.e. the arrangement of tokens, is not explicitly captured. We propose to cascade the recurrent neural networks to the Transformers, which referred to as the TransfoRNN model, to capture the sequential information. We found that the TransfoRNN models which consists of only shallow Transformers stack is suffice to give comparable, if not better, performance than a deeper Transformer model. Evaluated on the Penn Treebank and WikiText-2 corpora, the proposed TransfoRNN model has shown lower model perplexities with fewer number of model parameters. On the Penn Treebank corpus, the model perplexities were reduced up to 5.5% with the model size reduced up to 10.5%. On the WikiText-2 corpus, the model perplexity was reduced up to 2.2% with a 27.7% smaller model. Also, the TransfoRNN model was applied on the LibriSpeech speech recognition task and has shown comparable results with the Transformer models.
Fuzzy implication functions constructed from general overlap functions and fuzzy negations

Fuzzy implication functions have been widely investigated, both in theoretical and practical fields. The aim of this work is to continue previous works related to fuzzy implications constructed by means of non necessarily associative aggregation functions. In order to obtain a more general and flexible context, we extend the class of implications derived by fuzzy negations and t-norms, replacing the latter by general overlap functions. We also investigate their properties, characterization and intersections with other classes of fuzzy implication functions.
Learning Mobile CNN Feature Extraction Toward Fast Computation of Visual Object Tracking

In this paper, we construct a lightweight, high-precision and high-speed object tracking using a trained CNN. Conventional methods with trained CNNs use VGG16 network which requires powerful computational resources. Therefore, there is a problem that it is difficult to apply in low computation resources environments. To solve this problem, we use MobileNetV3, which is a CNN for mobile terminals.Based on Feature Map Selection Tracking, we propose a new architecture that extracts effective features of MobileNet for object tracking. The architecture requires no online learning but only offline learning. In addition, by using features of objects other than tracking target, the features of tracking target are extracted more efficiently. We measure the tracking accuracy with Visual Tracker Benchmark and confirm that the proposed method can perform high-precision and high-speed calculation even in low computation resource environments.
Self-calibration technique for characterization of integrated THz waveguides

Emerging high-frequency accelerator technology in the terahertz regime is promising for the development of compact high-brightness accelerators and high resolution-power beam diagnostics. One resounding challenge when scaling to higher frequencies and to smaller structures is the proportional scaling of tolerances which can hinder the overall performance of the structure. Consequently, characterizing these structures is essential for nominal operation. Here, we present a novel and simple self-calibration technique to characterize the dispersion relation of integrated hollow THz-waveguides. The developed model is verified in simulation by extracting dispersion characteristics of a standard waveguide a priori known by theory. The extracted phase velocity does not deviate from the true value by more than 9 × 10 − 5 % . In experiments the method demonstrates its ability to measure dispersion characteristics of non-standard waveguides embedded with their couplers with an accuracy below ≈ 0 . 5 % and precision of ≈ 0 . 05 % . Equipped with dielectric lining the metallic waveguides act as slow wave structures, and the dispersion curves are compared without and with dielectric. A phase synchronous mode, suitable for transverse deflection, is found at 275 GHz .