Reviewerly/exHarmony huggingface dataset:
M<sup>3</sup>L: Language-based Video Editing via Multi-Modal Multi-Level Transformers

Video editing tools are widely used nowadays for digital design. Although the demand for these tools is high, the prior knowledge required makes it difficult for novices to get started. Systems that could follow natural language instructions to perform automatic editing would significantly improve accessibility. This paper introduces the language-based video editing (LBVE) task, which allows the model to edit, guided by text instruction, a source video into a target video. LBVE contains two features: 1) the scenario of the source video is preserved instead of generating a completely different video; 2) the semantic is presented differently in the target video, and all changes are controlled by the given instruction. We propose a Multi-Modal Multi-Level Transformer (M <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> L) to carry out LBVE. M <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> L dynamically learns the correspondence between video perception and language semantic at different levels, which benefits both the video understanding and video frame synthesis. We build three new datasets for evaluation, including two diagnostic and one from natural videos with human-labeled text. Extensive experimental results show that M <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> L is effective for video editing and that LBVE can lead to a new field toward vision-and-language research.
Disentangling Architecture and Training for Optical Flow

How important are training details and datasets to recent optical flow architectures like RAFT? And do they generalize? To explore these questions, rather than develop a new architecture, we revisit three prominent architectures, PWC-Net, IRR-PWC and RAFT, with a common set of modern training techniques and datasets, and observe significant performance gains, demonstrating the importance and generality of these training details. Our newly trained PWC-Net and IRR-PWC show surprisingly large improvements, up to 30% versus original published results on Sintel and KITTI 2015 benchmarks. Our newly trained RAFT obtains an Fl-all score of 4.31% on KITTI 2015 and an avg. rank of 1.7 for end-point error on Middlebury. Our results demonstrate the benefits of separating the contributions of architectures, training techniques and datasets when analyzing performance gains of optical flow methods. Our source code is available at https://autoflow-google.github.io .
Path Planning for Large-scale UAV Formation Based on Improved SA-APF Algorithm

An improved simulated annealing-artificial potential field (SA-APF) algorithm is proposed to solve the problem of large-scale dense unmanned aerial vehicle (UAV) formation path planning in three-dimensional space. First, the artificial potential field (APF) method is extended to three dimensions and combined with the kinematics of the drone to constrain the step length, a path that meets the kinematics of the drone is generated and all UAVs can reach the target synchronously. By designing a double-layer repulsive potential field, establishing a collision prediction and obstacle avoidance mechanism, the defects of the shock path of the traditional APF are overcome. In order to reduce the computational complexity of dense formation path planning, the simulated annealing (SA) algorithm is used to optimize the distance cost, and the optimal allocation is calculated to replace the initial task allocation for path planning. Finally, the improved SA-APF algorithm is used for simulations and flight experiments on 1,000 UAVs. The results show that the path generated by SA-APF is smoother and has lower complexity. The algorithm can greatly reduce resource consumption in actual application.
Portal Rendering and Creation Interactions in Virtual Reality

Transformative portals are a novel technique for supporting visualizations and interactions between locations that are not co-located in virtual reality (VR). We are interested in improving upon current portal rendering for VR, and supporting VR users to create new portals quickly and accurately. In this paper, we introduce a new high-level algorithm for rendering transformative portals in VR and present the benefits of our approach. By leveraging single-pass stereo and stencil portal rendering, we developed our algorithm to support multiple portals with “infinite” recursion, while running performant on mobile VR devices. We then benchmarked our performance against other common portal rendering implementations. In addition, we focused our research on VR interactions for handheld portals, in which the initial placement of the portal’s destination is imperative to its effectiveness. In this paper, we present four new portal creation interactions for handheld portals: fishing reel, raycast with fishing reel, marker, and projectile curve. Based on existing research, we also established standard controls for fine-grained portal manipulation. In a user study, we compared and evaluated how well the creation interactions performed and argue that projectile curve is the most suitable general-purpose technique.
Transformers as Meta-learners for Implicit Neural Representations

Implicit Neural Representations (INRs) have emerged and shown their benefits over discrete representations in recent years. However, fitting an INR to the given observations usually requires optimization with gradient descent from scratch, which is inefficient and does not generalize well with sparse observations. To address this problem, most of the prior works train a hypernetwork that generates a single vector to modulate the INR weights, where the single vector becomes an information bottleneck that limits the reconstruction precision of the output INR. Recent work shows that the whole set of weights in INR can be precisely inferred without the single-vector bottleneck by gradient-based meta-learning. Motivated by a generalized formulation of gradient-based meta-learning, we propose a formulation that uses Transformers as hypernetworks for INRs, where it can directly build the whole set of INR weights with Transformers specialized as set-to-set mapping. We demonstrate the effectiveness of our method for building INRs in different tasks and domains, including 2D image regression and view synthesis for 3D objects. Our work draws connections between the Transformer hypernetworks and gradient-based meta-learning algorithms and we provide further analysis for understanding the generated INRs. The project page with code is at https://yinboc.github.io/trans-inr/ .
Assessment and treatment of privacy issues in blockchain systems

The ability to capture and quantify any aspect of daily life via sensors, enabled by the Internet of Things ( IoT ), data have become one of the most important resources of the 21 st century. However, the high value of data also renders data an appealing target for criminals. Two key protection goals when dealing with data are therefore to maintain their permanent availability and to ensure their integrity. Blockchain technology provides a means of data protection that addresses both of these objectives. On that account, blockchains are becoming increasingly popular for the management of critical data. As blockchains are operated in a decentralized manner, they are not only protected against failures, but it is also ensured that neither party has sole control over the managed data. Furthermore, blockchains are immutable and tamper-proof data stores, whereby data integrity is guaranteed. While these properties are preferable from a data security perspective, they also pose a threat to privacy and confidentiality, as data cannot be concealed, rectified, or deleted once they are added to the blockchain. In this paper, we therefore investigate which features of the blockchain pose an inherent privacy threat when dealing with personal or confidential data. To this end, we consider to what extent blockchains are in compliance with applicable data protection laws, namely the European General Data Protection Regulation ( GDPR ). Based on our identified key issues, we assess which concepts and technical measures can be leveraged to address these issues in order to create a privacy-by-design blockchain system.
Data station

Pooling and sharing data increases and distributes its value. But since data cannot be revoked once shared, scenarios that require controlled release of data for regulatory, privacy, and legal reasons default to not sharing. Because selectively controlling what data to release is difficult, the few data-sharing consortia that exist are often built around data-sharing agreements resulting from long and tedious one-off negotiations. We introduce Data Station, a data escrow designed to enable the formation of data-sharing consortia. Data owners share data with the escrow knowing it will not be released without their consent. Data users delegate their computation to the escrow. The data escrow relies on delegated computation to execute queries without releasing the data first. Data Station leverages hardware enclaves to generate trust among participants, and exploits the centralization of data and computation to generate an audit log. We evaluate Data Station on machine learning and data-sharing applications while running on an untrusted intermediary. In addition to important qualitative advantages, we show that Data Station: i) outperforms federated learning baselines in accuracy and runtime for the machine learning application; ii) is orders of magnitude faster than alternative secure data-sharing frameworks; and iii) introduces small overhead on the critical path.
MiCS

Existing general purpose frameworks for gigantic model training, i.e., dense models with billions of parameters, cannot scale efficiently on cloud environment with various networking conditions due to large communication overheads. In this paper, we propose MiCS, which Minimizes the Communication Scale to bring down communication overhead. Specifically, by decreasing the number of participants in a communication collective, MiCS can utilize heterogeneous network bandwidth, reduce network traffic over slower links, reduce the latency of communications for maintaining high network bandwidth utilization, and amortize expensive global gradient synchronization overhead. Our evaluation on AWS shows that the system throughput of MiCS is up to 2.89× that of the state-of-the-art large model training systems. MiCS achieves near-linear scaling efficiency, which is up to 1.27× that of DeepSpeed. MiCS allows us to train a proprietary model with 100 billion parameters on 512 GPUs with 99.4% weak-scaling efficiency, and it is able to saturate over 54.5% theoretical computation power of each GPU on a public cloud with less GPU memory and more restricted networks than DGX-A100 clusters.
YOLM: A Remote Sensing Aircraft Detection Model

Aircraft detection in remote sensing images plays an essential role in military and civil applications. Many aircraft in remote sensing images are small targets and are covered by cloud and fog, which make it difficult to extract sufficient feature information for aircraft detection. In this paper, a single-stage object detection model YOLM (You can Look More) has achieved better remote sensing aircraft detection performance by extracting more features. YOLM includes an enhanced neck network with a four-layer feature pyramid and a path aggregation network. More layers enable the model to extract more detailed information. To use the background information around the aircraft as a supplement to aircraft features, an attention module which can obtain the image context information of aircraft is designed. Experiments are conducted on the aircraft subset of FAIR1M data set and some aircraft images we obtained, and the proposed model has out-performed many baseline methods such as YOLO V5s.
Mind the Gap in Distilling StyleGANs

StyleGAN family is one of the most popular Generative Adversarial Networks (GANs) for unconditional generation. Despite its impressive performance, its high demand on storage and computation impedes their deployment on resource-constrained devices. This paper provides a comprehensive study of distilling from the popular StyleGAN-like architecture. Our key insight is that the main challenge of StyleGAN distillation lies in the output discrepancy issue, where the teacher and student model yield different outputs given the same input latent code. Standard knowledge distillation losses typically fail under this heterogeneous distillation scenario. We conduct thorough analysis about the reasons and effects of this discrepancy issue, and identify that the mapping network plays a vital role in determining semantic information of generated images. Based on this finding, we propose a novel initialization strategy for the student model, which can ensure the output consistency to the maximum extent. To further enhance the semantic consistency between the teacher and student model, we present a latent-direction-based distillation loss that preserves the semantic relations in latent space. Extensive experiments demonstrate the effectiveness of our approach in distilling StyleGAN2 and StyleGAN3, outperforming existing GAN distillation methods by a large margin. Code is available at: https://github.com/xuguodong03/StyleKD .
Tackling Denial of Service Attacks on Key Management in Software-Defined Quantum Key Distribution Networks

A QKD network provides an additional security layer for IT-secure cryptographic key distribution that is added to existing conventional networks. Thus, QKD network components must be resilient to security challenges from conventional network environments. This paper provided solutions for realizing a Key Management System to avoid DoS attacks, and our solution allows applications to function securely in environments with a reduced amount of keys. In addition, we have provided approaches for allocating and managing QKD resources to avoid malicious key reservations. Simulation experiments verified the proposed solutions.
RM2Doc

Automatic generation of requirements documents is an essential feature of the model-driven CASE tools such as UML and SysML designers. However, the quality of the generated documents from the current tools highly depends on the attached descriptions of models but not the quality of the model itself. Besides, if the stockholders ask to generate ISO/IEC/IEEE 29148-2018 conformed documents, extra templates are required. In this paper, we propose a CASE tool named RM2Doc, which can automatically generate ISO/IEC/IEEE 29148-2018 conformed requirements documents from UML models without any templates. In addition, the flow description can be generated from a use case without additional information. Moreover, it can automatically generate the semantic description of system operations only based on the formal expression of OCL. We have conducted four case studies with over 50 use cases. Overall, the result is satisfactory. The 95% requirements documents can be generated from the requirements model without any human interactions in 1 second. The proposed tools can be further developed for the industry of software engineering.
Multiple Behaviors Recommendation with Graph Learning

Recommendation systems have been extensively investigated by existing studies. However, previous work only targets a single type of user behavior (e.g., purchase) while ignoring the fact that users usually have multiple behaviors when browsing products (e.g., view, click, add-to-cart). These different behaviors can generate a large number of attributes about users. Meanwhile, in typical collaborative filtering (CF) systems, users and items are generally handled separately. Therefore, the associativity between users and items is not taken into account. To solve the above problems, we propose a novel framework named as Multiple Behaviors recommendation with Graph Learning (MBGL). To capture multiple characteristics of users and items, we construct a knowledge graph with multiple entity relations between users and items from a variety of data, such as purchase, view, and add-to-cart. We apply the knowledge graph embedding (KGE) method to learn the pre-training embedding of entity and relation. To better learn high-hop embedding from multi-behavior data, we construct a heterogeneous user-item graph and further design relation-aware GAT to learn graph embedding with relation type. To support high-order dependency in the graph, we introduce the residual network to solve feature smoothing and vanishing gradient problems. Moreover, to obtain the high-quality user and item embedding, we design an attention fusion layer to learn the fusion embedding and adopt the multi-task learning to predict users' preferences under different behaviors. Experiment results on real-world dataset show that our model outperforms other recommendation methods.
2DSegFormer: 2-D Transformer Model for Semantic Segmentation on Aerial Images

2D position information of input tokens is essential for transformer-based semantic segmentation models, especially on high-resolution aerial images. However, recent transformer-based segmentation methods use the position encoding to record position information and most position encoding methods encode the 1D positions of tokens. Therefore, we propose a two-dimensional semantic transformer model (2DSegFormer) for semantic segmentation on aerial images. In 2DSegFormer, we design a novel 2D positional attention to accurately record the 2D position information required by the transformer. Furthermore, we design the dilated residual connection and use it instead of skip connection in the deep stages to get a larger receptive field. skip connections are used in the shallow stages of 2DSegFormer to pass the details to the corresponding stages in the decoder. Experimental results on UAVid, Vaihingen and AeroScapes data sets demonstrate the effectiveness of 2DSegFormer. Compared with the state-of-the-art methods, 2DSegFormer shows better performance and great robustness on three different data sets. In particular, 2DSegFormer-B2 achieves first place in the public ranking on the UAVid test set.
TransforMatcher: Match-to-Match Attention for Semantic Correspondence

Establishing correspondences between images remains a challenging task, especially under large appearance changes due to different viewpoints or intra-class variations. In this work, we introduce a strong semantic image matching learner, dubbed TransforMatcher, which builds on the success of transformer networks in vision domains. Un-like existing convolution- or attention-based schemes for correspondence, TransforMatcher performs global match-to-match attention for precise match localization and dynamic refinement. To handle a large number of matches in a dense correlation map, we develop a light-weight attention architecture to consider the global match-to-match interactions. We also propose to utilize a multi-channel correlation map for refinement, treating the multi-level scores as features instead of a single score to fully exploit the richer layer-wise semantics. In experiments, TransforMatcher sets a new state of the art on SPair-71k while performing on par with existing SOTA methods on the PF-PASCAL dataset.
Supporting Massive DLRM Inference through Software Defined Memory

Deep Learning Recommendation Models (DLRM) are widespread, account for a considerable data center footprint, and grow by more than 1.5x per year. With model size soon to be in terabytes range, leveraging Storage Class Memory (SCM) for inference enables lower power consumption. This paper evaluates the major challenges in extending the memory hierarchy to SCM for DLRM, and presents different techniques to improve performance through a Software Defined Memory. We show how underlying technologies such as Nand Flash and 3DXP differentiate, and relate to real world scenarios, enabling from 5% to 29% power savings.
Metasurface-Based Optical Analog Computing: From Fundamentals to Applications

With unprecedented growth in data information over the past few decades, it is critically important to seek a faster and more efficient method to simplify data processing. In recent years, optical analog computing provides an available way due to its large capacity, energy saving, and high efficiency. Metasurfaces, as 2-dimensional artificial nanostructures, have shown an extraordinary ability for controlling light and paving a pathway toward all-optical analog computing. In this brief review, we discuss the latest development in metasurface-based optical analog computing. Theoretical fundamentals and experimental demonstrations of optical analog computing for resolving a variety of mathematical problems have been reviewed, including differential operation, convolution operation, and Fourier transformation. Finally, a brief introduction of emerging applications in microscopy imaging, quantum imaging, and holographic imaging is presented.
Application of Augmented Reality for Equipment Maintenance and Employee Training in Manufacturing Plant

The industrial design development of AR devices provides a convenient way to use AR technology for production, training, and inspection. In this paper, the application of augmented reality technology is explored for industrial purposes through the professional engines, Unity and Vuforia SDK, for Android app development. A manufacturing plant is used as an example, and preventive equipment maintenance is achieved by an instruction presentation on a display. By using AR technology, new employees with average ability and little experience can accurately perform various on-site maintenance operations, which helps learn technical experience.
VAMYOLOX: An Accurate and Efficient Object Detection Algorithm Based on Visual Attention Mechanism for UAV Optical Sensors

Unmanned aerial vehicles (UAVs) have been widely used in various fields. As one of the key technologies in improving the autonomous sensing ability of UAV optical sensors, object detection has become a research focus in recent years. Since UAVs usually navigate at different vertical heights, the object scales and sensor field of view change violently, which brings a great difficulty to the optimization of the model. Moreover, when a UAV is flying at low level rapidly, it may cause the motion blur phenomenon on objects that are highly dense in position, leading to great challenge for distinction of these objects. To address the extremely tough problems discussed above, we propose an accurate and efficient object detection algorithm, namely VAMYOLOX. Based on YOLOX, we first redesigned the classification and regression loss function of the model to better conduct classification and localization under complex motion blur and dense scenes. Then we increase another prediction head to detect lots of tiny objects to ultimately improve the detection ability of the model for multi-scale objects. Finally, we redesigned the neck of the network by introducing the triplet attention module (TAM) to find attention regions in scenes with dense objects and in images that cover a large area, accordingly enhancing the features extracted by the backbone network. Extensive experiments on dataset VisDrone widely used in the research of UAV image object detection show that VAMYOLOX has achieved state-of-the-art (SOTA) performance with good interpretability in UAV optical sensors captured scenes. On VisDrone-DET-test-dev subset, the average precision (AP) of VAMYOLOX is 25.31%, outperforming the previous SOTA model (CornerNet) by 1.88%. On VisDrone-DET-val subset, the AP of our method is 29.4%, achieving highly competitive result with previous SOTA method (AMRNet). Not only that, VAMYOLOX achieves a maximum improvement of 2.72% compared to the AP of the baseline model (YOLOX). In addition, compared with other methods, our method has a significant advantage in speed and can meet the needs of different scenarios. The PyTorch code and trained models are available at https://github.com/yangyahu-1994/VAMYOLOX.