OpenRaiser/Intern-Atlas huggingface dataset:
Asynchronous Wireless Federated Learning with Probabilistic Client Selection
Venue: arXiv cs.LG 2023
Year: 2023

Federated learning (FL) is a promising distributed learning framework where distributed clients collaboratively train a machine learning model coordinated by a server. To tackle the stragglers issue in asynchronous FL, we consider that each client keeps local updates and probabilistically transmits the local model to the server at arbitrary times. We first derive the (approximate) expression for the convergence rate based on the probabilistic client selection. Then, an optimization problem is formulated to trade off the convergence rate of asynchronous FL and mobile energy consumption by joint probabilistic client selection and bandwidth allocation. We develop an iterative algorithm to solve the non-convex problem globally optimally. Experiments demonstrate the superiority of the proposed approach compared with the traditional schemes.
DI-Net : Decomposed Implicit Garment Transfer Network for Digital Clothed 3D Human
Venue: arXiv cs.CV 2023
Year: 2023

3D virtual try-on enjoys many potential applications and hence has attracted wide attention. However, it remains a challenging task that has not been adequately solved. Existing 2D virtual try-on methods cannot be directly extended to 3D since they lack the ability to perceive the depth of each pixel. Besides, 3D virtual try-on approaches are mostly built on the fixed topological structure and with heavy computation. To deal with these problems, we propose a Decomposed Implicit garment transfer network (DI-Net), which can effortlessly reconstruct a 3D human mesh with the newly try-on result and preserve the texture from an arbitrary perspective. Specifically, DI-Net consists of two modules: 1) A complementary warping module that warps the reference image to have the same pose as the source image through dense correspondence learning and sparse flow learning; 2) A geometry-aware decomposed transfer module that decomposes the garment transfer into image layout based transfer and texture based transfer, achieving surface and texture reconstruction by constructing pixel-aligned implicit functions. Experimental results show the effectiveness and superiority of our method in the 3D virtual try-on task, which can yield more high-quality results over other existing methods.
Compressing the Backward Pass of Large-Scale Neural Architectures by Structured Activation Pruning
Venue: arXiv cs.LG 2023
Year: 2023

The rise of Deep Neural Networks (DNNs) has led to an increase in model size and complexity, straining the memory capacity of GPUs. Sparsity in DNNs, characterized as structural or ephemeral, has gained attention as a solution. This work focuses on ephemeral sparsity, aiming to reduce memory consumption during training. It emphasizes the significance of activations, an often overlooked component, and their role in memory usage. This work employs structured pruning in Block Sparse Compressed Row (BSR) format in combination with a magnitude-based criterion to efficiently prune activations. We furthermore introduce efficient block-sparse operators for GPUs and showcase their effectiveness, as well as the superior compression offered by block sparsity. We report the effectiveness of activation pruning by evaluating training speed, accuracy, and memory usage of large-scale neural architectures on the example of ResMLP on image classification tasks. As a result, we observe a memory reduction of up to 32% while maintaining accuracy. Ultimately, our approach aims to democratize large-scale model training, reduce GPU requirements, and address ecological concerns.
Technological Challenges of Ambient Serious Games in Higher Education
Venue: arXiv cs.HC 2023
Year: 2023

Naturally, university courses should be designed to attract students, engaging them to achieve learning goals. Toward this end, the use of Serious Games has been proposed in the literature. To address positive effects, such as content memorability and attendance rates, we propose Ambient Serious Games as games embedded in a computer-enriched environment, which is only partially perceived mentally by players. In this paper, we describe five technological key challenges that must be overcome to seamlessly and beneficially integrate an Ambient Serious Game into teaching. These challenges, derived from a scenario, focus on the technological provision and conduct of such games based on a software platform. They include (1) the integration of physical smart learning objects in heterogeneous environments under dynamic constraints, (2) the representation of abstract subject matter using smart learning objects, (3) the guided or automatic connection of all involved components, (4) the explanation of the components, their interaction, as well as the serious game itself, and (5) feedback on the game state.
Dendrogram distance: an evaluation metric for generative networks using hierarchical clustering
Venue: arXiv cs.LG 2023
Year: 2023

We present a novel metric for generative modeling evaluation, focusing primarily on generative networks. The method uses dendrograms to represent real and fake data, allowing for the divergence between training and generated samples to be computed. This metric focus on mode collapse, targeting generators that are not able to capture all modes in the training set. To evaluate the proposed method it is introduced a validation scheme based on sampling from real datasets, therefore the metric is evaluated in a controlled environment and proves to be competitive with other state-of-the-art approaches.
HumanRef: Single Image to 3D Human Generation via Reference-Guided Diffusion
Venue: arXiv cs.CV 2023
Year: 2023

Generating a 3D human model from a single reference image is challenging because it requires inferring textures and geometries in invisible views while maintaining consistency with the reference image. Previous methods utilizing 3D generative models are limited by the availability of 3D training data. Optimization-based methods that lift text-to-image diffusion models to 3D generation often fail to preserve the texture details of the reference image, resulting in inconsistent appearances in different views. In this paper, we propose HumanRef, a 3D human generation framework from a single-view input. To ensure the generated 3D model is photorealistic and consistent with the input image, HumanRef introduces a novel method called reference-guided score distillation sampling (Ref-SDS), which effectively incorporates image guidance into the generation process. Furthermore, we introduce region-aware attention to Ref-SDS, ensuring accurate correspondence between different body regions. Experimental results demonstrate that HumanRef outperforms state-of-the-art methods in generating 3D clothed humans with fine geometry, photorealistic textures, and view-consistent appearances.
DemoFusion: Democratising High-Resolution Image Generation With No $$$
Venue: arXiv cs.CV 2023
Year: 2023

High-resolution image generation with Generative Artificial Intelligence (GenAI) has immense potential but, due to the enormous capital investment required for training, it is increasingly centralised to a few large corporations, and hidden behind paywalls. This paper aims to democratise high-resolution GenAI by advancing the frontier of high-resolution generation while remaining accessible to a broad audience. We demonstrate that existing Latent Diffusion Models (LDMs) possess untapped potential for higher-resolution image generation. Our novel DemoFusion framework seamlessly extends open-source GenAI models, employing Progressive Upscaling, Skip Residual, and Dilated Sampling mechanisms to achieve higher-resolution image generation. The progressive nature of DemoFusion requires more passes, but the intermediate results can serve as "previews", facilitating rapid prompt iteration.
Counter-terrorism in cyber-physical spaces: Best practices and technologies from the state of the art
Venue: arXiv cs.CY 2023
Year: 2023

Context: The demand for protection and security of physical spaces and urban areas increased with the escalation of terroristic attacks in recent years. We envision with the proposed cyber-physical systems and spaces, a city that would indeed become a smarter urbanistic object, proactively providing alerts and being protective against any threat. Objectives: This survey intend to provide a systematic multivocal literature survey comprised of an updated, comprehensive and timely overview of state of the art in counter-terrorism cyber-physical systems, hence aimed at the protection of cyber-physical spaces. Hence, provide guidelines to law enforcement agencies and practitioners providing a description of technologies and best practices for the protection of public spaces. Methods: We analyzed 112 papers collected from different online sources, both from the academic field and from websites and blogs ranging from 2004 till mid-2022. Results: a) There is no one single bullet-proof solution available for the protection of public spaces. b) From our analysis we found three major active fields for the protection of public spaces: Information Technologies, Architectural approaches, Organizational field. c) While the academic suggest best practices and methodologies for the protection of urban areas, the market did not provide any type of implementation of such suggested approaches, which shows a lack of fertilization between academia and industry. Conclusion: The overall analysis has led us to state that there is no one single solution available, conversely, multiple methods and techniques can be put in place to guarantee safety and security in public spaces. The techniques range from architectural design to rethink the design of public spaces keeping security into account in continuity, to emerging technologies such as AI and predictive surveillance.
LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models
Venue: arXiv cs.CV 2023
Year: 2023

In this work, we present a novel method to tackle the token generation challenge in Vision Language Models (VLMs) for video and image understanding, called LLaMA-VID. Current VLMs, while proficient in tasks like image captioning and visual question answering, face computational burdens when processing long videos due to the excessive visual tokens. LLaMA-VID addresses this issue by representing each frame with two distinct tokens, namely context token and content token. The context token encodes the overall image context based on user input, whereas the content token encapsulates visual cues in each frame. This dual-token strategy significantly reduces the overload of long videos while preserving critical information. Generally, LLaMA-VID empowers existing frameworks to support hour-long videos and pushes their upper limit with an extra context token. It is proved to surpass previous methods on most of video- or image-based benchmarks. Code is available https://github.com/dvlab-research/LLaMA-VID}{https://github.com/dvlab-research/LLaMA-VID
ClimateX: Do LLMs Accurately Assess Human Expert Confidence in Climate Statements?
Venue: arXiv cs.LG 2023
Year: 2023

Evaluating the accuracy of outputs generated by Large Language Models (LLMs) is especially important in the climate science and policy domain. We introduce the Expert Confidence in Climate Statements (ClimateX) dataset, a novel, curated, expert-labeled dataset consisting of 8094 climate statements collected from the latest Intergovernmental Panel on Climate Change (IPCC) reports, labeled with their associated confidence levels. Using this dataset, we show that recent LLMs can classify human expert confidence in climate-related statements, especially in a few-shot learning setting, but with limited (up to 47%) accuracy. Overall, models exhibit consistent and significant over-confidence on low and medium confidence statements. We highlight implications of our results for climate communication, LLMs evaluation strategies, and the use of LLMs in information retrieval systems.
LSTM model predicting outcome of strategic thinking task exhibits representations of level-k thinking
Venue: arXiv cs.GT 2023
Year: 2023

Which neural mechanisms underlie strategic thinking in the human brain? Neuroeconomic research has not yet bridged the gap between theoretical models of higher-order reasoning and the precise mechanisms implemented in neural networks in the human brain. In this paper, I demonstrate that a recurrent neural network model can learn to perform strongly in the simple strategic game Rock-Paper-Scissors. In doing so, it develops implicit representations of strategically important variables (the levels $k$ of reasoning) which economists have postulated in theoretical models. These representations can be extracted from the hidden activations of the neural network. These findings hint at a connection between the mechanisms implicit in recurrent neural networks and models of strategic thinking in economic theory. Future empirical brain research can investigate whether these mechanisms correspond to mechanisms implicit in biological neural networks.
ReWaRD: Retinal Waves for Pre-Training Artificial Neural Networks Mimicking Real Prenatal Development
Venue: arXiv cs.CV 2023
Year: 2023

Computational models trained on a large amount of natural images are the state-of-the-art to study human vision - usually adult vision. Computational models of infant vision and its further development are gaining more and more attention in the community. In this work we aim at the very beginning of our visual experience - pre- and post-natal retinal waves which suggest to be a pre-training mechanism for the primate visual system at a very early stage of development. We see this approach as an instance of biologically plausible data driven inductive bias through pre-training. We built a computational model that mimics this development mechanism by pre-training different artificial convolutional neural networks with simulated retinal wave images. The resulting features of this biologically plausible pre-training closely match the V1 features of the primate visual system. We show that the performance gain by pre-training with retinal waves is similar to a state-of-the art pre-training pipeline. Our framework contains the retinal wave generator, as well as a training strategy, which can be a first step in a curriculum learning based training diet for various models of development. We release code, data and trained networks to build the basis for future work on visual development and based on a curriculum learning approach including prenatal development to support studies of innate vs. learned properties of the primate visual system. An additional benefit of our pre-trained networks for neuroscience or computer vision applications is the absence of biases inherited from datasets like ImageNet.
Pattern retrieval of traffic congestion using graph-based associations of traffic domain-specific features
Venue: arXiv cs.CV 2023
Year: 2023

The fast-growing amount of traffic data brings many opportunities for revealing more insightful information about traffic dynamics. However, it also demands an effective database management system in which information retrieval is arguably an important feature. The ability to locate similar patterns in big datasets potentially paves the way for further valuable analyses in traffic management. This paper proposes a content-based retrieval system for spatiotemporal patterns of highway traffic congestion. There are two main components in our framework, namely pattern representation and similarity measurement. To effectively interpret retrieval outcomes, the paper proposes a graph-based approach (relation-graph) for the former component, in which fundamental traffic phenomena are encoded as nodes and their spatiotemporal relationships as edges. In the latter component, the similarities between congestion patterns are customizable with various aspects according to user expectations. We evaluated the proposed framework by applying it to a dataset of hundreds of patterns with various complexities (temporally and spatially). The example queries indicate the effectiveness of the proposed method, i.e. the obtained patterns present similar traffic phenomena as in the given examples. In addition, the success of the proposed approach directly derives a new opportunity for semantic retrieval, in which expected patterns are described by adopting the relation-graph notion to associate fundamental traffic phenomena.
Reducing energy consumption of cloud data centers using proper placement of virtual machines
Venue: arXiv cs.NI 2023
Year: 2023

In today's world, the use of cloud data centers for easy access to data and processing resources is expanding rapidly. Rapid technology growth and increasing number of users make hardware and software architectures upgrade a constant need. The necessary infrastructure to implement this architecture is the use of virtual machines in physical systems. The main issue in this architecture is how to allocate virtual machines to physical machines on the network. In this paper we have proposed a method to use virtualization for minimizing energy consumption and decreasing the cloud resource waste. We have used learning automata as a reinforcement learning model for optimal placement of virtual machines. The simulation results show the proposed method has good performance in reducing energy consumption of servers in cloud data centers.
Digital Twins for Logistics and Supply Chain Systems: Literature Review, Conceptual Framework, Research Potential, and Practical Challenges
Venue: arXiv cs.SE 2023
Year: 2023

To facilitate an effective, efficient, transparent, and timely decision-making process as well as to provide guidelines for industry planning and public policy development, a conceptual framework of digital twins (DTs) for logistics and supply chain systems (LSCS) is needed. This paper first introduces the background of the logistics and supply chain industry, the DT and its potential benefits, and the motivations and scope of this research. The literature review indicates research and practice gaps and needs that motivate proposing a new conceptual DT framework for LSCS. As each element of the new framework has different requirements and goals, it initiates new research opportunities and creates practical implementation challenges. As such, the future of DT computation involves advanced analytics and modeling techniques to address the new agenda's requirements. Finally, ideas on the next steps to deploy a transparent, trustworthy, and resilient DT for LSCS are presented.
Impact of Indoor Mobility Behavior on the Respiratory Infectious Diseases Transmission Trends
Venue: arXiv cs.CY 2023
Year: 2023

The importance of indoor human mobility in the transmission dynamics of respiratory infectious diseases has been acknowledged. Previous studies have predominantly addressed a single type of mobility behavior such as queueing and a series of behaviors under specific scenarios. However, these studies ignore the abstraction of mobility behavior in various scenes and the critical examination of how these abstracted behaviors impact disease propagation. To address these problems, this study considers people's mobility behaviors in a general scenario, abstracting them into two main categories: crowding behavior, related to the spatial aspect, and stopping behavior, related to the temporal aspect. Accordingly, this study investigates their impacts on disease spreading and the impact of individual spatio-temporal distribution resulting from these mobility behaviors on epidemic transmission. First, a point of interest (POI) method is introduced to quantify the crowding-related spatial POI factors (i.e., the number of crowdings and the distance between crowdings) and stopping-related temporal POI factors (i.e., the number of stoppings and the duration of each stopping). Besides, a personal space determined with Voronoi diagrams is used to construct the individual spatio-temporal distribution factor. Second, two indicators (i.e., the daily number of new cases and the average exposure risk of people) are applied to quantify epidemic transmission. These indicators are derived from a fundamental model which accurately predicts disease transmission between moving individuals. Third, a set of 200 indoor scenarios is constructed and simulated to help determine variable values. Concurrently, the influences and underlying mechanisms of these behavioral factors on disease transmission are examined using structural equation modeling and causal inference modeling......
How does spatial structure affect psychological restoration? A method based on Graph Neural Networks and Street View Imagery
Venue: arXiv cs.CV 2023
Year: 2023

The Attention Restoration Theory (ART) presents a theoretical framework with four essential indicators (being away, extent, fascinating, and compatibility) for comprehending urban and natural restoration quality. However, previous studies relied on non-sequential data and non-spatial dependent methods, which overlooks the impact of spatial structure defined here as the positional relationships between scene entities on restoration quality. The past methods also make it challenging to measure restoration quality on an urban scale. In this work, a spatial-dependent graph neural networks (GNNs) approach is proposed to reveal the relation between spatial structure and restoration quality on an urban scale. Specifically, we constructed two different types of graphs at the street and city levels. The street-level graphs, using sequential street view images (SVIs) of road segments to capture position relationships between entities, were used to represent spatial structure. The city-level graph, modeling the topological relationships of roads as non-Euclidean data structures and embedding urban features (including Perception-features, Spatial-features, and Socioeconomic-features), was used to measure restoration quality. The results demonstrate that: 1) spatial-dependent GNNs model outperforms traditional methods (Acc = 0.735, F1 = 0.732); 2) spatial structure portrayed through sequential SVIs data significantly influences restoration quality; 3) spaces with the same restoration quality exhibited distinct spatial structures patterns. This study clarifies the association between spatial structure and restoration quality, providing a new perspective to improve urban well-being in the future.
Asynchronous Merkle Trees
Venue: arXiv cs.DC 2023
Year: 2023

Merkle trees have become a widely successful cryptographic data structure. Enabling a vast variety of applications from checking for inconsistencies in databases like Dynamo to essential tools like Git to large scale distributed systems like Bitcoin and other blockchains. There have also been various versions of Merkle trees like Jellyfish Merkle Trees and Sparse Merkle Trees designed for different applications. However, one key drawback of all these Merkle trees is that with a large data set the cost of computing the tree increases significantly, moreover insert operations on a single leaf require re-building the entire tree. For certain use cases building the tree this way is acceptable, however in environments where compute time needs to be as low as possible and where data is processed in parallel, we are presented with a need for asynchronous computation. This paper proposes a solution where given a batch of data that has to be processed concurrently, a Merkle Tree can be constructed from the batch asynchronously without needing to recalculate the tree for every insert.