City brain: practice of large‐scale artificial intelligence in the real world

IET Smart CitiesVolume 1, Issue 1 p. 28-37 Research ArticleOpen Access City brain: practice of large-scale artificial intelligence in the real world Jianfeng Zhang, Jianfeng Zhang DAMO Academy, Alibaba Group, 969 West Wenyi Road, Hangzhou, Zhejiang Province, People's Republic of ChinaSearch for more papers by this authorXian-Sheng Hua, Corresponding Author Xian-Sheng Hua xiansheng.hxs@alibaba-inc.com DAMO Academy, Alibaba Group, 969 West Wenyi Road, Hangzhou, Zhejiang Province, People's Republic of ChinaSearch for more papers by this authorJianqiang Huang, Jianqiang Huang DAMO Academy, Alibaba Group, 969 West Wenyi Road, Hangzhou, Zhejiang Province, People's Republic of ChinaSearch for more papers by this authorXu Shen, Xu Shen DAMO Academy, Alibaba Group, 969 West Wenyi Road, Hangzhou, Zhejiang Province, People's Republic of ChinaSearch for more papers by this authorJingyuan Chen, Jingyuan Chen DAMO Academy, Alibaba Group, 969 West Wenyi Road, Hangzhou, Zhejiang Province, People's Republic of ChinaSearch for more papers by this authorQin Zhou, Qin Zhou DAMO Academy, Alibaba Group, 969 West Wenyi Road, Hangzhou, Zhejiang Province, People's Republic of ChinaSearch for more papers by this authorZhihang Fu, Zhihang Fu DAMO Academy, Alibaba Group, 969 West Wenyi Road, Hangzhou, Zhejiang Province, People's Republic of ChinaSearch for more papers by this authorYiru Zhao, Yiru Zhao DAMO Academy, Alibaba Group, 969 West Wenyi Road, Hangzhou, Zhejiang Province, People's Republic of China Shanghai Jiao Tong University, 800 Dongchuan Road, Shanghai, People's Republic of ChinaSearch for more papers by this author Jianfeng Zhang, Jianfeng Zhang DAMO Academy, Alibaba Group, 969 West Wenyi Road, Hangzhou, Zhejiang Province, People's Republic of ChinaSearch for more papers by this authorXian-Sheng Hua, Corresponding Author Xian-Sheng Hua xiansheng.hxs@alibaba-inc.com DAMO Academy, Alibaba Group, 969 West Wenyi Road, Hangzhou, Zhejiang Province, People's Republic of ChinaSearch for more papers by this authorJianqiang Huang, Jianqiang Huang DAMO Academy, Alibaba Group, 969 West Wenyi Road, Hangzhou, Zhejiang Province, People's Republic of ChinaSearch for more papers by this authorXu Shen, Xu Shen DAMO Academy, Alibaba Group, 969 West Wenyi Road, Hangzhou, Zhejiang Province, People's Republic of ChinaSearch for more papers by this authorJingyuan Chen, Jingyuan Chen DAMO Academy, Alibaba Group, 969 West Wenyi Road, Hangzhou, Zhejiang Province, People's Republic of ChinaSearch for more papers by this authorQin Zhou, Qin Zhou DAMO Academy, Alibaba Group, 969 West Wenyi Road, Hangzhou, Zhejiang Province, People's Republic of ChinaSearch for more papers by this authorZhihang Fu, Zhihang Fu DAMO Academy, Alibaba Group, 969 West Wenyi Road, Hangzhou, Zhejiang Province, People's Republic of ChinaSearch for more papers by this authorYiru Zhao, Yiru Zhao DAMO Academy, Alibaba Group, 969 West Wenyi Road, Hangzhou, Zhejiang Province, People's Republic of China Shanghai Jiao Tong University, 800 Dongchuan Road, Shanghai, People's Republic of ChinaSearch for more papers by this author First published: 24 June 2019 https://doi.org/10.1049/iet-smc.2019.0034Citations: 13AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract A city is an aggregate of a huge amount of heterogeneous data. However, extracting meaningful values from that data remains a challenge. City Brain is an end-to-end system whose goal is to glean irreplaceable values from big city data, specifically from videos, with the assistance of rapidly evolving artificial intelligence technologies and fast-growing computing capacity. From cognition to optimisation, to decision-making, from search to prediction and ultimately, to intervention, City Brain improves the way to manage the city, as well as the way to live in it. In this study, the authors introduce current practices of the City Brain platform in a few cities in China, including what they can do to achieve the goal and make it a reality. Then they focus on the system overview and key technical details of each component of the City Brain system, from cognition to intervention. Lastly, they present a few deployment cases of City Brain in various cities in China. 1 Introduction 1.1 About City Brain As early as 2016, Smart City was presented as a national strategy in China: We should profoundly understand the role of the Internet in nation management and society governance, taking the implementation of e-government and building new smart cities as the key points. We will build a nationally integrated big data center by data integration and promote technology convergence, business integration, and data convergence to achieve collaborative management and services across geographies, systems, departments, and services. Today, the first batch of 'Digital Twin Cities' using artificial intelligence (AI) technologies have realised the Internet mode of data sharing, data co-creation, and data automatic control with the help of Alibaba City Brain. The City Brain is the 'commanding heights' of technologies in Alibaba Group. Based on the elastic calculation and large-scale data processing platform of Alibaba Cloud, integrated with the top capabilities of interdisciplinary fields such as machine vision, large-scale topological network computing, and traffic flow analysis, the City Brain is capable of massive multi-source data collection, real-time processing, and intelligent computing. There are three metrics for a real 'City Brain': (1) it can deal with ultra-large-scale and multi-source data that humans cannot understand in real time (global cognition); (2) It can understand the complex hidden rules that humans have not discovered (machine learning); (3) It can formulate a global optimal strategy that surpasses local suboptimal decision made by human (global coordination). The City Brain has become a powerful assistant for city managers in cognising, transforming, and operating cities. It transcends human capabilities with four kinds of 'super powers': (1) machine vision cognitive capability to enhance perception of urban data; (2) the full-scale data platform construction capacity to enhance the 'data density' and 'particle management' level; (3) real-time computing capability under large-scale dynamic topology networks; (4) the City Brain open platform capability to empower the digital city industry. The City Brain is deployed according to five major application scenarios: urban traffic checkup, urban police monitoring, urban traffic micro-control, urban special vehicles, and urban strategic planning. (1) Urban traffic checkup can completely quantify the urban 'vital signs' via the fusion and integration of full-scale, full-network, and cross-domain data, avoiding one-sided solutions for urban problems due to the single source data; (2) By taking advantage of machine learning and computer vision, automatic police monitoring can liberate police officers from laborious legwork, and let the data to run errands, instead of police officers; (3) Urban traffic micro control-and-feedback loop. It opens the feedback control system between 'brains', 'eyes', and 'hands and feet'. Based on multi-source data, the global intelligent algorithm provides a fine-grained control of city-scale traffic signals to improve mobility in the city; (4) Route optimisation for emergency vehicles. City Brain identifies the quickest route for emergency vehicles to arrive at the scene within the shortest time frame; (5) Urban layout planning and verification, which analyses the effect of a proposed urban construction blueprint on the cloud with the simulation data model. 1.2 History In April 2016, the concept of 'city brain' was formally proposed. City Brain is a new infrastructure built on massive data, which utilises AI to solve urban governance and development issues that cannot be solved by the human brain. It is a program that offers a comprehensive suite of acquisition, integration, and analysis of big and heterogeneous data generated by a diversity of sources in urban spaces through video and image recognition, data mining and machine learning technology. With this, city council and urban planners will be able to make better decisions for the community. In November 2017, Alibaba Cloud ET City Brain was selected as one of the first four AI innovation platforms by the Ministry of Science and Technology, which became a major contribution of Chinese technology to the world's urban area. On January 29, 2018, the Malaysia Digital Economy Corp (MDEC) and the Dewan Bandaraya Kuala Lumpur (DBKL) jointly announced the introduction of Alibaba Cloud ET City Brain. The AI will be fully applied to Malaysia's traffic management, urban planning, and environmental protection. It is the first time that the City Brain went out to serve worldwide customers. It has been three years from the birth of City Brain to the present. The City Brain has been launched in Hangzhou, Shanghai, Chongqing, Suzhou, Haikou, Beijing, Chengdu, Quzhou, Jiaxing, Kuala Lumpur, Macao, and many other cities. 2 Overview of the City Brain In this project, the challenges we are facing are all about three keywords: cost, value, and difference. Whether the cost for such a big computation, storage, and network intensive task is manageable, whether the technology is ready to get the values from those data, and whether the values are sufficiently significant. What has been challenged even more is that where are the differences compared with 'video surveillance' and 'edge computing'. These questions can be well answered by taking a closer look at the City Brain (Fig. 1). First, we have a bunch of data from the city, including the video data. The first step is to acquire the data and understand the data. We call this step 'Cognition', which includes recognising what is on the road and what is happening on the road, such as the cars, the people, the cyclist, the traffic status, the accidents etc [[1]]. Fig. 1Open in figure viewerPowerPoint 100 feet view of the City Brain Then, the second step 'Decision and Optimisation', we make decisions or optimise the ways we run the city based on the cognitive results, e.g. automatic accident alerting [[2]], traffic light optimisation. Thereafter, in the 'Search and Mining' step, we put everything the cameras have seen into a database and build an index, thus we can apply search on this data. For example, we find a suspicious car or discover patterns in the data, such as finding the root cause of traffic congestion somewhere in the city [[3]]. Next, based on current and historical data, we can predict what is going to happen next, either in a short period of time, such as the traffic congestion possibility after 20 min for an intersection, or next day's accident possibility of a road section, given the weather condition and event information of the city. Last, based on predicted results, resources can be pre-allocated to respond to those situations more effectively. For example, if we know the possibility of accident will increase three times given the bad weather tomorrow as well as a few events that will gather a large number of people, we can adjust the traffic lights and send traffic advice to prevent those bad things from happening. We call this 'Prediction' and 'Intervention'. In the remaining part of this paper, we will present more details about the aforementioned parts, as well as the specifically designed large-scale visual computing platform. 3 Large-scale visual computing platform 3.1 System overview With the rapid development of urbanisation, large amount of video data is generated every day in a city. These videos play critical roles in city management, public safety, traffic control, and environment protection etc. However, video data is unstructured. How to effectively store, analyse, and further take advantage of these videos has been a worldwide problem. In order to address the above problem, our team builds the large-scale visual computing platform to meet the requirement for real-time, comprehensive, large-scale smart video analysis, which makes joint perception, prediction, alarm and prevention in smart city management possible for the first time. The overall architecture of the platform is illustrated in Fig. 2, which composes of three core systems, namely 'the Access and Transmitting system', 'the Computing system', and 'the Searching system'. The access and transmitting stage perform data accessing, data pre-processing, data resource scheduling, data transmitting, and video streaming. Fig. 2Open in figure viewerPowerPoint Architecture of the large-scale visual computing platform Based on the stream-processing framework (Flink [[4]]), the computing system has the following key capabilities: batch computing, stream computing, model parallelisation, model scheduling, graphical calculation, and atlas calculation. These key techniques are able to support the top-level applications such as online/offline video analysis, trajectory tracking, feature quantitation etc. The searching system consists of the large-scale search engine, online feature extraction service, and search strategy engine. The search engine performs real-time index compression. Online feature extraction is responsible for extracting features of the city objects from video frames. The search strategy engine links the former two modules and provides an image search service to target customers. The visual computing platform can be deployed on the cloud. It could be shared and reused through the cloud resource pool, fully exploiting the efficiency of multi-core and ensuring elastic expansion. Besides, by means of the peak staggered multiplexing, the platform achieves flexible and efficient resource utilisation. The distributed deployment of cloud host could provide intelligent analysis capability on demand, thus improving the efficiency of intelligent analysis. With the large-scale visual computing platform, we provide the capabilities of AI, large-scale data processing and cloud computing to the upper-level application layer, allowing customers to focus on business innovation. 3.2 Key technical details 3.2.1 Distributed heterogeneous scheduling engine The large-scale video computing resource scheduling system manages the cloud video computing resources and dynamically adjusts the resource allocation to best utilise the computing ability [[4], [5]]. Its core functions include single-node heterogeneous computing scheduling, distributed heterogeneous computing resource scheduling, and distributed task dynamic allocation. Single-node heterogeneous computing scheduling: this part evaluates the model's requirements for computing resources, and allocates appropriate heterogeneous computing resources (central processing unit, graphics processing unit etc.) and model operating parameters to the model on a single node according to the actual configurations of the machines. In this way, we can improve the resource utilisation rate as well as the number of video streams that can be processed on a single node. Distributed Heterogeneous Computing Resource Scheduling: this part analyses and evaluates the computing resources for all tasks running on the streaming computing platform and allocates different tasks to various computing nodes according to the composition of heterogeneous computing resources and the resource requirements of different tasks. By using the cloud resource pool to share and reuse, the multi-core efficiency can be fully utilised to ensure flexible expansion, thus improving the utilisation of heterogeneous resources of the entire cluster and finally reducing the energy consumption of the entire cluster. Dynamic allocation of distributed tasks: Due to changes in time periods and scenarios, resource requirements for different tasks may change dramatically across time and space. Distributed task dynamic allocation performs real-time statistical analysis on the running status of tasks and effectively redistributes these tasks. Flexible and efficient resource utilisation can be achieved through peak staggered multiplexing. 3.2.2 Graph computation Traditional video analysis systems mainly focus on recognising certain objects within frames, which is far from enough for scene perception. To fully understand the scene, we need not only to recognise each separate object, but also analyse the relationships among these objects. Towards the end, the scene graph is designed to model the relationships within objects. Thereafter, the graph can be indexed and retrieved to support upper layer applications such as searching and prediction based on the scene graph. To achieve this goal, our large-scale visual computing platform is designed to support the functions of graph indexing [[6]] and graph searching, which will be detailed in the following sections. Graph Indexing and Graph query: graph indexing is a very important pre-processing step in graph query. Indexing guarantees the uniqueness of each row of data in the database table. Besides, it can greatly speed up the retrieval of data, which is the main reason for creating an index. However, creating and maintaining an index takes extra time and physical space, which increases the maintenance cost of the data. To enable query based on graph data, the large-scale visual computing platform adopts the state-of-the-art index and search techniques. By taking the relationships among the graph nodes into consideration, we can make globally optimised predictions and interventions on the real-time city events. 3.2.3 Model quantisation and acceleration To efficiently execute deep models on the proposed large-scale visual computing platform, we introduce network quantitation techniques to reduce the computation load [[7]]. Our work is devoted to quantising full-precision networks into low-bit networks. Existing methods formulate the low-bit quantisation of networks as an approximation or optimisation problem. Approximation-based methods confront the gradient mismatch problem, while optimisation-based methods are only suitable for quantising weights and can introduce high computational cost during the training stage. In our large-scale visual computing platform, we provide a simple and uniform way for weights and activations quantisation by formulating it as a differentiable non-linear function. As shown in Fig. 3, the quantisation function is formed as a linear combination of several Sigmoid functions with learnable biases and scales. In this way, the proposed quantisation function can be learned in a lossless and end-to-end manner and works for any weights and activations in neural networks, thereby avoiding the gradient mismatch problem. It can further be trained via continuous relaxation of the steepness of the Sigmoid functions (shown in Fig. 4). Fig. 3Open in figure viewerPowerPoint Quantisation function for a neural network Fig. 4Open in figure viewerPowerPoint Relaxation process of a quantisation function during training, which goes from a straight line to steps as the temperature T increases (a) No quantisation, (b) T = 1, (c) T = 11, (d) T = 121, (e) Complete quantisation 4 Cognition 4.1 System overview City management involves a lot of data resources. Video data, with its intuitive, mass, and real-time characteristics, is an important part of the data resources of the city. The traditional way of city patrolling mainly relies on laborious manual monitoring. In contrast, through the processing and analysis of massive video, the cognition system can not only obtain the running status of the urban public area in real time, but also detect abnormal events in specific areas in time. According to the architecture shown in Fig. 5, the system consists of three main stages: visual data access stage, multimedia processing stage, and visual algorithm application stage. Fig. 5Open in figure viewerPowerPoint Architecture of the cognition system in City Brain In the visual data access stage, video resources from different manufacturers are accessed through standard video protocols. The system has the ability to access large-scale video data based on the cloud platform, which meets the demand of comprehensive city cognition. The accessed data includes online video streams, offline video files, and static images, which will further be preprocessed and transcoded at the multimedia processing stage. In the multimedia processing stage, visual data is transmitted to the system through the local area network of a city. The large-scale video and images are decoded, transcoded, or preprocessed in this stage. Furthermore, this stage also collects parameters of video sources including camera position and alarm configurations to comprehensively manage multimedia information. In the visual algorithm application stage, the all-time all-area cognition system integrates fundamental tasks such as image recognition, object detection, object tracking, scene recognition, and anomaly detection. These tasks are formed into independent modules to support top-level algorithm applications. Specifically, traffic accident perception integrates image recognition, object detection, and object tracking tasks. The road congestion perception involves object detection and object tracking tasks. The sudden violence event perception is based on scene recognition and anomaly detection tasks. And the object detection and anomaly detection tasks are utilised to achieve the alarming of persons and vehicles in restricted area. Based on the aforementioned rich top-level visual algorithm applications, the system is further applied to a variety of public scenes in the city, such as transportation, subway, campus, and community. 4.2 Key technical details The all-time all-area city cognition system pursues a precise understanding of a variety of scenarios. It recognises what is on the road and what is happening on the road before making decisions or alarms. In this section, we will introduce our object detection and anomaly detection methods deployed in this system. 4.2.1 Object detection and tracking Object detection is one of the core tasks in cognition problems. In the cognition system, detecting objects on the road, such as vehicles and pedestrians, is the primary step for perception applications. Therefore, the high accuracy of the detection algorithm is a prerequisite for subsequent applications. We have devoted great efforts in object detection research. For vehicle detection, we proposed a scheme, which is illustrated in Fig. 6, based on multi-task deep convolutional neural networks (CNN), region-of-interest (RoI) voting, and multi-level localisation, denoted by RV-CNN [[1]]. In the design of CNN architecture, we enriched the supervised information with subcategory, region overlap, bounding-box regression, and category of each training RoI as a multi-task learning framework. This design allows the CNN model to share visual knowledge among different vehicle attributes simultaneously, and thus, detection robustness can be effectively improved. We introduced the subcategory classification task to enforce the CNN model to learn a good representation for vehicles under different occlusions, truncations, and viewpoints. In addition, we utilised the CNN model to predict the offset direction of each RoI boundary toward the corresponding ground truth. Then, each RoI could vote those suitable adjacent bounding boxes, which are consistent with this additional information. For clarity, suppose a predicted box has coordinates and score s. And we denote its neighbouring RoIs by B, the number of RoIs in B by N and the i th RoI with assigned score and predicted directions by . Then we formulate the voting scheme as (1) in which (2) Fig. 6Open in figure viewerPowerPoint Illustration of RV-CNN multi-task framework. RoI pooling layer is proposed to extract features for each RoI. Then the pooled features are used for category classification, bounding box regression, overlap prediction, and subcategory classification Other functions follow the same rule as . After the scores of all predicted boxes are computed again. The voting results are combined with the score of each RoI itself to find a more accurate location from a large number of candidates. For pedestrian detection, we introduced a previewer block [[8]] which previews the objectness probability for the potential regression region of each prior box, using the stronger features with larger receptive fields and more contextual information for better predictions. The proposed previewer blocks preselect regions with high confidences containing objects by involving enough contextual information. The detector then classifies and relocates the prior boxes in these regions. In addition, we introduced a new metric intersection of ground-truth (IoG) ratio to formulate the containment relations between the previewer region and ground-truth bounding boxes. (3) where N is the number of ground-truth objects. An object is completely contained by the previewer region when IoG = 1.0, and we assign a positive label to this region. A previewer region will get a negative label if IoG < 0.8. Furthermore, the label of a larger region which contains an object is set to be ignored (neither positive nor negative during training) when that object is already contained in smaller previewer region. With the previewer blocks, plenty of small-scale false positives were eliminated during the inference process and we've got an effective performance on pedestrian detection. Besides, we use the renowned kernelised correlation filters [[9]] for multiple objects tracking based on object detection results. Object tracking effectively maps the corresponding detected objects between different frames. Combined with object detection, object tracking module first illustrates the trajectories of vehicles and pedestrians over a period of time, and then identifies target behaviours. 4.2.2 Event detection Anomalous events detection in real-world video scenes is a challenging problem due to the complexity of 'anomaly' as well as the cluttered backgrounds, objects and motions in the scenes. Most existing methods use hand-crafted features in local spatial regions to identify anomalies. We proposed a Spatio-Temporal AutoEncoder (ST AutoEncoder or STAE) [[2]], which utilises deep neural networks to learn video representation automatically and extracts features from both spatial and temporal dimensions by performing three-dimensional (3D) convolutions. Fig. 7 shows the details of the framework: an encoder followed by two branches of decoder for reconstructing past frames and predicting future frames, respectively. Fig. 7Open in figure viewerPowerPoint Architecture of the network. An encoder followed by two branches of decoder for reconstructing past frames and predicting future frames, respectively In addition to the reconstruction loss used in existing typical autoencoders, we introduced a weight-decreasing prediction loss for generating future frames, which enhances the motion feature learning in videos. Specifically, the reconstruction branch and the prediction branch share the same hidden feature layer but perform different tasks: reconstructing the past sequence and predicting the future sequence, respectively. The prediction task guides the model to capture the trajectory of moving objects and enforce the encoder to better extract the temporal features. The prediction loss is formulated by: (4) where is the input hyper-cuboid, is the output of the prediction branch, is the ground truth of the future T frames and the superscript t in is the frame of the video clip X. The frame has a weight of , which decreases as t increases. With the anomaly detection framework, the all-time all-area patrolling and alerting system can detect abnormal events in a variety of scenarios in real time, and then notify the city manager in the form of alarms. Real-time alarm anomaly events through video surveillance can help the government officials to quickly detect and even prevent abnormal emergencies, ensuring the public safety and operation efficiency of a city. 5 Decision and optimisation 5.1 System overview Based on the acquisition, integration, and analysis of big and heterogeneous data generated by a diversity of sources in urban spaces, the City Brain can optimise the flow of vehicles and traffic signals, and upgrade the city governance and decision-making on traffic command and road construction. The whole decision and optimisation system are depicted in Fig. 8, which consists of three main stages: the data perceptron stage, the data fusion stage, the decision and optimisation stage. Fig. 8Open in figure viewerPowerPoint Architecture of the decision and optimisation system in the City Brain In the data perceptron stage, data from various sources in urban spaces and departments are collected and analysed. First is the video data, including general video streams and bayonet camera streams. Traffic accidents (collision, jam etc.) and traffic parameters (road traffic flow, traffic light status, traffic volume and speed in particular lanes etc.) are generated from these video streams. For map data, high-definition map with road network topology, origin-destination dat