In this task, you are given the abstract of a research paper. Your task is to generate a summary of this abstract. Your summary should not be very short, but it's better if it's not more than 30 words.

Multi-task learning (MTL) with neural networks leverages commonalities in tasks to improve performance, but often suffers from task interference which reduces the benefits of transfer. To address this issue we introduce the routing network paradigm, a novel neural network and training algorithm. A routing network is a kind of self-organizing neural network consisting of two components: a router and a set of one or more function blocks. A function block may be any neural network – for example a fully-connected or a convolutional layer. Given an input the router makes a routing decision, choosing a function block to apply and passing the output back to the router recursively, terminating when a fixed recursion depth is reached. In this way the routing network dynamically composes different function blocks for each input. We employ a collaborative multi-agent reinforcement learning (MARL) approach to jointly train the router and function blocks. We evaluate our model against cross-stitch networks and shared-layer baselines on multi-task settings of the MNIST, mini-imagenet, and CIFAR-100 datasets. Our experiments demonstrate a significant improvement in accuracy, with sharper convergence. In addition, routing networks have nearly constant per-task training cost while cross-stitch networks scale linearly with the number of tasks. On CIFAR100 (20 tasks) we obtain cross-stitch performance levels with an 85% average reduction in training time.

routing networks: a new kind of neural network which learns to adaptively route its input for multi-task learning

Researches on deep neural networks with discrete parameters and their deployment in embedded systems have been active and promising topics. Although previous works have successfully reduced precision in inference, transferring both training and inference processes to low-bitwidth integers has not been demonstrated simultaneously. In this work, we develop a new method termed as ``"WAGE" to discretize both training and inference, where weights (W), activations (A), gradients (G) and errors (E) among layers are shifted and linearly constrained to low-bitwidth integers. To perform pure discrete dataflow for fixed-point devices, we further replace batch normalization by a constant scaling layer and simplify other components that are arduous for integer implementation. Improved accuracies can be obtained on multiple datasets, which indicates that WAGE somehow acts as a type of regularization. Empirically, we demonstrate the potential to deploy training in hardware systems such as integer-based deep learning accelerators and neuromorphic chips with comparable accuracy and higher energy efficiency, which is crucial to future AI applications in variable scenarios with transfer and continual learning demands.
We apply training and inference with only low-bitwidth integers in DNNs

Successful training of convolutional neural networks is often associated with suffi-
 ciently deep architectures composed of high amounts of features. These networks
 typically rely on a variety of regularization and pruning techniques to converge
 to less redundant states. We introduce a novel bottom-up approach to expand
 representations in fixed-depth architectures. These architectures start from just a
 single feature per layer and greedily increase width of individual layers to attain
 effective representational capacities needed for a specific task. While network
 growth can rely on a family of metrics, we propose a computationally efficient
 version based on feature time evolution and demonstrate its potency in determin-
 ing feature importance and a networks’ effective capacity. We demonstrate how
 automatically expanded architectures converge to similar topologies that benefit
 from lesser amount of parameters or improved accuracy and exhibit systematic
 correspondence in representational complexity with the specified task. In contrast
 to conventional design patterns with a typical monotonic increase in the amount of
 features with increased depth, we observe that CNNs perform better when there is
 more learnable parameters in intermediate, with falloffs to earlier and later layers.

A bottom-up algorithm that expands CNNs starting with one feature per layer to architectures with sufficient representational capacity.