OpenRaiser/Intern-Atlas huggingface dataset:
Assessment of CFD capability for prediction of the Coandă effect
Venue: arXiv cs.CE 2021
Year: 2021

The tendency of a jet to stay attached to a flat or convex surface is called the Coandă effect and has many potential technical applications. The aim of this thesis is to assess how well Computational Fluid Dynamics can capture it. A Reynolds-Averaged Navier-Stokes approach with a 2-dimensional domain was first used to simulate an offset jet on a flat plane. Whether it was for $k-ω$ SST or $k-ε$ turbulence model, a good prediction of the flow was found. Since it is known that streamline curvature can have an important impact on the numerical results, a jet blown tangentially to a cylinder was then considered. Using the same approach as for the flat plane, with $k-ω$ SST turbulence model, some of the flow features such as the separation location or velocity profiles near the jet exit were accurately predicted. However, the jet development was overall poorly captured. A Curvature Correction was then introduced in the turbulence model and if it did slightly improve the jet development, the negative impact on other quantities makes its benefits questionable. Due to the known presence of longitudinal and spanwise vortices in the flow, a Reynolds-Averaged Navier-Stokes approach with a 3-dimensional domain was attempted but was only able to reproduce the 2-dimensional results. Although if the longitudinal vortices are artificially generated at the inlet, their development is supported by the simulation. Finally, since the shortcomings of the numerical results obtained might be due to limitations of the Reynolds-Averaged Navier-Stokes approach, a Large Eddy Simulation was attempted. Unfortunately, due to time and computational restrictions a fully developed flow could not be obtained, the methodology and preliminary results are however presented.
The Typed Graph Model -- a Supermodel for Model Management and Data Integration
Venue: arXiv cs.DB 2021
Year: 2021

In recent years, the Graph Model has become increasingly popular, especially in the application domain of social networks. The model has been semantically augmented with properties and labels attached to the graph elements. It is difficult to ensure data quality for the properties and the data structure because the model does not need a schema. In this paper, we propose a schema bound Typed Graph Model with properties and labels. These enhancements improve not only data quality but also the quality of graph analysis. The power of this model is provided by using hyper-nodes and hyper-edges, which allows to present data structures on different abstraction levels. We prove that the model is at least equivalent in expressive power to most popular data models. Therefore, it can be used as a supermodel for model management and data integration. We illustrate by example the superiority of this model over the property graph data model of Hidders and other prevalent data models, namely the relational, object-oriented, XML model, and RDF Schema.
Exploiting Twitter as Source of Large Corpora of Weakly Similar Pairs for Semantic Sentence Embeddings
Venue: arXiv cs.CL 2021
Year: 2021

Semantic sentence embeddings are usually supervisedly built minimizing distances between pairs of embeddings of sentences labelled as semantically similar by annotators. Since big labelled datasets are rare, in particular for non-English languages, and expensive, recent studies focus on unsupervised approaches that require not-paired input sentences. We instead propose a language-independent approach to build large datasets of pairs of informal texts weakly similar, without manual human effort, exploiting Twitter's intrinsic powerful signals of relatedness: replies and quotes of tweets. We use the collected pairs to train a Transformer model with triplet-like structures, and we test the generated embeddings on Twitter NLP similarity tasks (PIT and TURL) and STSb. We also introduce four new sentence ranking evaluation benchmarks of informal texts, carefully extracted from the initial collections of tweets, proving not only that our best model learns classical Semantic Textual Similarity, but also excels on tasks where pairs of sentences are not exact paraphrases. Ablation studies reveal how increasing the corpus size influences positively the results, even at 2M samples, suggesting that bigger collections of Tweets still do not contain redundant information about semantic similarities.
Structured Prediction in NLP -- A survey
Venue: arXiv cs.CL 2021
Year: 2021

Over the last several years, the field of Structured prediction in NLP has had seen huge advancements with sophisticated probabilistic graphical models, energy-based networks, and its combination with deep learning-based approaches. This survey provides a brief of major techniques in structured prediction and its applications in the NLP domains like parsing, sequence labeling, text generation, and sequence to sequence tasks. We also deep-dived into energy-based and attention-based techniques in structured prediction, identified some relevant open issues and gaps in the current state-of-the-art research, and have come up with some detailed ideas for future research in these fields.
Exploring the Limits of Large Scale Pre-training
Venue: arXiv cs.LG 2021
Year: 2021

Recent developments in large-scale machine learning suggest that by scaling up data, model size and training time properly, one might observe that improvements in pre-training would transfer favorably to most downstream tasks. In this work, we systematically study this phenomena and establish that, as we increase the upstream accuracy, the performance of downstream tasks saturates. In particular, we investigate more than 4800 experiments on Vision Transformers, MLP-Mixers and ResNets with number of parameters ranging from ten million to ten billion, trained on the largest scale of available image data (JFT, ImageNet21K) and evaluated on more than 20 downstream image recognition tasks. We propose a model for downstream performance that reflects the saturation phenomena and captures the nonlinear relationship in performance of upstream and downstream tasks. Delving deeper to understand the reasons that give rise to these phenomena, we show that the saturation behavior we observe is closely related to the way that representations evolve through the layers of the models. We showcase an even more extreme scenario where performance on upstream and downstream are at odds with each other. That is, to have a better downstream performance, we need to hurt upstream accuracy.
Self-Supervised Generative Style Transfer for One-Shot Medical Image Segmentation
Venue: arXiv cs.CV 2021
Year: 2021

In medical image segmentation, supervised deep networks' success comes at the cost of requiring abundant labeled data. While asking domain experts to annotate only one or a few of the cohort's images is feasible, annotating all available images is impractical. This issue is further exacerbated when pre-trained deep networks are exposed to a new image dataset from an unfamiliar distribution. Using available open-source data for ad-hoc transfer learning or hand-tuned techniques for data augmentation only provides suboptimal solutions. Motivated by atlas-based segmentation, we propose a novel volumetric self-supervised learning for data augmentation capable of synthesizing volumetric image-segmentation pairs via learning transformations from a single labeled atlas to the unlabeled data. Our work's central tenet benefits from a combined view of one-shot generative learning and the proposed self-supervised training strategy that cluster unlabeled volumetric images with similar styles together. Unlike previous methods, our method does not require input volumes at inference time to synthesize new images. Instead, it can generate diversified volumetric image-segmentation pairs from a prior distribution given a single or multi-site dataset. Augmented data generated by our method used to train the segmentation network provide significant improvements over state-of-the-art deep one-shot learning methods on the task of brain MRI segmentation. Ablation studies further exemplified that the proposed appearance model and joint training are crucial to synthesize realistic examples compared to existing medical registration methods. The code, data, and models are available at https://github.com/devavratTomar/SST.
FacialFilmroll: High-resolution multi-shot video editing
Venue: arXiv cs.CV 2021
Year: 2021

We present FacialFilmroll, a solution for spatially and temporally consistent editing of faces in one or multiple shots. We build upon unwrap mosaic [Rav-Acha et al. 2008] by specializing it to faces. We leverage recent techniques to fit a 3D face model on monocular videos to (i) improve the quality of the mosaic for edition and (ii) permit the automatic transfer of edits from one shot to other shots of the same actor. We explain how FacialFilmroll is integrated in post-production facility. Finally, we present video editing results using FacialFilmroll on high resolution videos.
Rate Splitting Multiple Access for Semi-Grant-Free Transmissions
Venue: arXiv cs.IT 2021
Year: 2021

Enabled by hybrid grant-based (GB) and grant-free (GF) transmission techniques, GF users of internet of things (IoT) devices and massive machine-type communications (mMTC) meet opportunities to share wireless resources with GB users. In this paper, we propose a rate splitting multiple access (RSMA) strategy for an emerging semi-grant-free (SGF) transmission system to increase connectivity and reliability. In the proposed RSMA assisted SGF (RSMA-SGF) scheme, the GF users apply the rate splitting principle to realize distributed contentions and utilize transmit power most effectively for robust transmissions, meanwhile keeping themselves transparent to the GB user. Compared to existing non-orthogonal multiple access (NOMA) assisted SGF schemes, the RSMA-SGF scheme significantly decreases outage probability and achieves full multiuser diversity gain without restricting the GB and GF users' target rates to a limited value region. Exact expressions and asymptotic analysis for the outage probability are provided to facilitate the system performance evaluation of the proposed RSMA-SGF scheme. Computer simulation results clarify the superior outage performance of the RSMA-SGF scheme and verify the accuracy of the developed analytical results.
Machine learning attack on copy detection patterns: are 1x1 patterns cloneable?
Venue: arXiv cs.CR 2021
Year: 2021

Nowadays, the modern economy critically requires reliable yet cheap protection solutions against product counterfeiting for the mass market. Copy detection patterns (CDP) are considered as such solution in several applications. It is assumed that being printed at the maximum achievable limit of a printing resolution of an industrial printer with the smallest symbol size 1x1 elements, the CDP cannot be copied with sufficient accuracy and thus are unclonable. In this paper, we challenge this hypothesis and consider a copy attack against the CDP based on machine learning. The experimental based on samples produced on two industrial printers demonstrate that simple detection metrics used in the CDP authentication cannot reliably distinguish the original CDP from their fakes. Thus, the paper calls for a need of careful reconsideration of CDP cloneability and search for new authentication techniques and CDP optimization because of the current attack.
Secure Aggregation for Buffered Asynchronous Federated Learning
Venue: arXiv cs.LG 2021
Year: 2021

Federated learning (FL) typically relies on synchronous training, which is slow due to stragglers. While asynchronous training handles stragglers efficiently, it does not ensure privacy due to the incompatibility with the secure aggregation protocols. A buffered asynchronous training protocol known as FedBuff has been proposed recently which bridges the gap between synchronous and asynchronous training to mitigate stragglers and to also ensure privacy simultaneously. FedBuff allows the users to send their updates asynchronously while ensuring privacy by storing the updates in a trusted execution environment (TEE) enabled private buffer. TEEs, however, have limited memory which limits the buffer size. Motivated by this limitation, we develop a buffered asynchronous secure aggregation (BASecAgg) protocol that does not rely on TEEs. The conventional secure aggregation protocols cannot be applied in the buffered asynchronous setting since the buffer may have local models corresponding to different rounds and hence the masks that the users use to protect their models may not cancel out. BASecAgg addresses this challenge by carefully designing the masks such that they cancel out even if they correspond to different rounds. Our convergence analysis and experiments show that BASecAgg almost has the same convergence guarantees as FedBuff without relying on TEEs.
Securing Federated Learning: A Covert Communication-based Approach
Venue: arXiv cs.CR 2021
Year: 2021

Federated Learning Networks (FLNs) have been envisaged as a promising paradigm to collaboratively train models among mobile devices without exposing their local privacy data. Due to the need for frequent model updates and communications, FLNs are vulnerable to various attacks (e.g., eavesdropping attacks, inference attacks, poisoning attacks, and backdoor attacks). Balancing privacy protection with efficient distributed model training is a key challenge for FLNs. Existing countermeasures incur high computation costs and are only designed for specific attacks on FLNs. In this paper, we bridge this gap by proposing the Covert Communication-based Federated Learning (CCFL) approach. Based on the emerging communication security technique of covert communication which hides the existence of wireless communication activities, CCFL can degrade attackers' capability of extracting useful information from the FLN training protocol, which is a fundamental step for most existing attacks, and thereby holistically enhances the privacy of FLNs. We experimentally evaluate CCFL extensively under real-world settings in which the FL latency is optimized under given security requirements. Numerical results demonstrate the significant effectiveness of the proposed approach in terms of both training efficiency and communication security.
Transparent Forwarders: An Unnoticed Component of the Open DNS Infrastructure
Venue: arXiv cs.NI 2021
Year: 2021

In this paper, we revisit the open DNS (ODNS) infrastructure and, for the first time, systematically measure and analyze transparent forwarders, DNS components that transparently relay between stub resolvers and recursive resolvers. Our key findings include four takeaways. First, transparent forwarders contribute 26% (563k) to the current ODNS infrastructure. Unfortunately, common periodic scanning campaigns such as Shadowserver do not capture transparent forwarders and thus underestimate the current threat potential of the ODNS. Second, we find an increased deployment of transparent forwarders in Asia and South America. In India alone, the ODNS consists of 80% transparent forwarders. Third, many transparent forwarders relay to a few selected public resolvers such as Google and Cloudflare, which confirms a consolidation trend of DNS stakeholders. Finally, we introduce DNSRoute++, a new traceroute approach to understand the network infrastructure connecting transparent forwarders and resolvers.
On the Impact of Stable Ranks in Deep Nets
Venue: arXiv cs.LG 2021
Year: 2021

A recent line of work has established intriguing connections between the generalization/compression properties of a deep neural network (DNN) model and the so-called layer weights' stable ranks. Intuitively, the latter are indicators of the effective number of parameters in the net. In this work, we address some natural questions regarding the space of DNNs conditioned on the layers' stable rank, where we study feed-forward dynamics, initialization, training and expressivity. To this end, we first propose a random DNN model with a new sampling scheme based on stable rank. Then, we show how feed-forward maps are affected by the constraint and how training evolves in the overparametrized regime (via Neural Tangent Kernels). Our results imply that stable ranks appear layerwise essentially as linear factors whose effect accumulates exponentially depthwise. Moreover, we provide empirical analysis suggesting that stable rank initialization alone can lead to convergence speed ups.
How to Query An Oracle? Efficient Strategies to Label Data
Venue: arXiv cs.LG 2021
Year: 2021

We consider the basic problem of querying an expert oracle for labeling a dataset in machine learning. This is typically an expensive and time consuming process and therefore, we seek ways to do so efficiently. The conventional approach involves comparing each sample with (the representative of) each class to find a match. In a setting with $N$ equally likely classes, this involves $N/2$ pairwise comparisons (queries per sample) on average. We consider a $k$-ary query scheme with $k\ge 2$ samples in a query that identifies (dis)similar items in the set while effectively exploiting the associated transitive relations. We present a randomized batch algorithm that operates on a round-by-round basis to label the samples and achieves a query rate of $O(\frac{N}{k^2})$. In addition, we present an adaptive greedy query scheme, which achieves an average rate of $\approx 0.2N$ queries per sample with triplet queries. For the proposed algorithms, we investigate the query rate performance analytically and with simulations. Empirical studies suggest that each triplet query takes an expert at most 50\% more time compared with a pairwise query, indicating the effectiveness of the proposed $k$-ary query schemes. We generalize the analyses to nonuniform class distributions when possible.
Approximate $\mathrm{CVP}$ in time $2^{0.802 \, n}$ -- now in any norm!
Venue: arXiv cs.DS 2021
Year: 2021

We show that a constant factor approximation of the shortest and closest lattice vector problem in any norm can be computed in time $2^{0.802\, n}$. This contrasts the corresponding $2^n$ time, (gap)-SETH based lower bounds for these problems that even apply for small constant approximation. For both problems, $\mathrm{SVP}$ and $\mathrm{CVP}$, we reduce to the case of the Euclidean norm. A key technical ingredient in that reduction is a twist of Milman's construction of an $M$-ellipsoid which approximates any symmetric convex body $K$ with an ellipsoid $\mathcal{E}$ so that $2^{\varepsilon n}$ translates of a constant scaling of $\mathcal{E}$ can cover $K$ and vice versa.
Explaining Off-Policy Actor-Critic From A Bias-Variance Perspective
Venue: arXiv cs.LG 2021
Year: 2021

Off-policy Actor-Critic algorithms have demonstrated phenomenal experimental performance but still require better explanations. To this end, we show its policy evaluation error on the distribution of transitions decomposes into: a Bellman error, a bias from policy mismatch, and a variance term from sampling. By comparing the magnitude of bias and variance, we explain the success of the Emphasizing Recent Experience sampling and 1/age weighted sampling. Both sampling strategies yield smaller bias and variance and are hence preferable to uniform sampling.
Pedestrian Wind Factor Estimation in Complex Urban Environments
Venue: arXiv cs.LG 2021
Year: 2021

Urban planners and policy makers face the challenge of creating livable and enjoyable cities for larger populations in much denser urban conditions. While the urban microclimate holds a key role in defining the quality of urban spaces today and in the future, the integration of wind microclimate assessment in early urban design and planning processes remains a challenge due to the complexity and high computational expense of computational fluid dynamics (CFD) simulations. This work develops a data-driven workflow for real-time pedestrian wind comfort estimation in complex urban environments which may enable designers, policy makers and city residents to make informed decisions about mobility, health, and energy choices. We use a conditional generative adversarial network (cGAN) architecture to reduce the computational computation while maintaining high confidence levels and interpretability, adequate representation of urban complexity, and suitability for pedestrian comfort estimation. We demonstrate high quality wind field approximations while reducing computation time from days to seconds.
More on Change-Making and Related Problems
Venue: arXiv cs.DS 2021
Year: 2021

Given a set of $n$ integer-valued coin types and a target value $t$, the well-known change-making problem asks for the minimum number of coins that sum to $t$, assuming an unlimited number of coins in each type. In the more general all-targets version of the problem, we want the minimum number of coins summing to $j$, for every $j=0,\ldots,t$. For example, the textbook dynamic programming algorithms can solve the all-targets problem in $O(nt)$ time. Recently, Chan and He (SOSA'20) described a number of $O(t\,\textrm{polylog}\,t)$-time algorithms for the original (single-target) version of the change-making problem, but not the all-targets version. We obtain a number of new results on change-making and related problems, including: 1. A new algorithm for the all-targets change-making problem with running time $\tilde{O}(t^{4/3})$, improving a previous $\tilde{O}(t^{3/2})$-time algorithm. 2. A very simple $\tilde{O}(u^2+t)$-time algorithm for the all-targets change-making problem, where $u$ denotes the maximum coin value. The analysis of the algorithm uses a theorem of Erdős and Graham (1972) on the Frobenius problem. This algorithm can be extended to solve the all-capacities version of the unbounded knapsack problem (for integer item weights bounded by $u$). 3. For the original (single-target) coin changing problem, we describe a simple modification of one of Chan and He's algorithms that runs in $\tilde{O}(u)$ time (instead of $\tilde{O}(t)$). 4. For the original (single-capacity) unbounded knapsack problem, we describe a simple algorithm that runs in $\tilde{O}(nu)$ time, improving previous near-$u^2$-time algorithms. 5. We also observe how one of our ideas implies a new result on the minimum word break problem, an optimization version of a string problem studied by Bringmann et al. (FOCS'17), generalizing change-making (which corresponds to the unary special case).
KNN-BERT: Fine-Tuning Pre-Trained Models with KNN Classifier
Venue: arXiv cs.CL 2021
Year: 2021

Pre-trained models are widely used in fine-tuning downstream tasks with linear classifiers optimized by the cross-entropy loss, which might face robustness and stability problems. These problems can be improved by learning representations that focus on similarities in the same class and contradictions in different classes when making predictions. In this paper, we utilize the K-Nearest Neighbors Classifier in pre-trained model fine-tuning. For this KNN classifier, we introduce a supervised momentum contrastive learning framework to learn the clustered representations of the supervised downstream tasks. Extensive experiments on text classification tasks and robustness tests show that by incorporating KNNs with the traditional fine-tuning process, we can obtain significant improvements on the clean accuracy in both rich-source and few-shot settings and can improve the robustness against adversarial attacks. \footnote{all codes is available at https://github.com/LinyangLee/KNN-BERT}