UEC e-Bulletin
Updates on research, innovation, and events at UEC:
Unique and Exciting Campus in Tokyo

Vol.26, Feburary 2021

The University of Electro-Communications 1-5-1 Chofugaoka, Chofu, Tokyo 182-8585© 2021 The University of Electro-Communications.

Vol.26, March 2021

Research Highlights
 Innovative automated control systems: Control-theoretic approach for fast online reinforcement
learning
 Computing in close proximity: Edge intelligence with deep reinforcement learning

Researcher Video Profiles
 Innovative control theory: Bridging the gap between research on control, reinforcement learning,and power systems

News and Events UEC holds the 7th UEC Seminar in ASEAN, 2020 and the 2nd ASEAN - UEC Workshop

The University of Electro-Communications 1-5-1 Chofugaoka, Chofu, Tokyo 182-8585© 2021 The University of Electro-Communications.

Vol.26, March 2021

Innovative automated control systems: Control-theoretic approach for fast online
reinforcement learning

Reinforcement Learning (RL) is an effective way of designing model-free linear quadratic regulators(LQRs) for linear time-invariant networks with unknown state-space models. RL has wide ranging
applications including industrial automation, self-driving automobiles, power grid systems, and even
forecasting stock prices for financial markets.
However, conventional RL can result in unacceptably long learning times when network sizes arelarge. This can pose a serious challenge for real-time decision-making.
Tomonori Sadamoto at the University of Electro-Communications, Aranya Chakrabortty at North
Carolina State University, USA, and Jun-ichi Imura at the Tokyo Institute of Technology haveproposed a fast RL algorithm that enables online control of large-scale network systems.
Their approach is to construct a compressed state vector by projecting the measured state through aprojection matrix. This matrix is constructed from online measurements of the states in a way that it
captures the dominant controllable subspace of the open-loop network model. Next, a RL-controller
is learned using the reduced-dimensional state instead of the original state such that the resultant costis close to the optimal LQR cost.
The lower dimensionality of this approach enables a drastic reduction in the computational
complexity for learning. Moreover, stability and optimality of the control performance aretheoretically evaluated using robust control theory by treating the dimensionality-reduction error asan uncertainty. Numerical simulations through a 100-dimensional large-scale power grid model
showed that the learning speed improved by almost 23 times while maintaining control performance.
The main contribution of the paper is to show how two individually well-known concepts indynamical system theory and machine learning, namely, model reduction and reinforcement
learning, can be combined to construct a highly efficient real-time control design for extreme-scale
networks.

The University of Electro-Communications 1-5-1 Chofugaoka, Chofu, Tokyo 182-8585© 2021 The University of Electro-Communications.

Vol.26, March 2021

Reference・ Authors: Tomonori Sadamoto, Aranya Chakrabortty, and Jun-ichi Imura・ Title of original paper: Fast Online Reinforcement Learning Control using State-SpaceDimensionality Reduction
・ IEEE Transactions on Control of Network Systems (Early Access)・ Digital Object Identifier (DOI): 10.1109/TCNS.2020.3027780
・ Affiliations: Department of Mechanical Engineering and Intelligent Systems, Graduate School ofInformatics and Engineering, The University of Electro-Communications.
・ First Author website: http://www.sc.lab.uec.ac.jp/ts/index.html

Figure caption: Transient response of frequency deviation of generators in the power system(a)without control, (b) by the optimal controller designed by an existing RL method, and (c) by thecontroller designed by the proposed algorithm.

The University of Electro-Communications 1-5-1 Chofugaoka, Chofu, Tokyo 182-8585© 2021 The University of Electro-Communications.

Vol.26, March 2021

Computing in close proximity: Edge intelligence with deep reinforcement learning

Mobile edge computing (MEC) is a promising paradigm to improve the quality of computation
experience for mobile devices by providing computing capabilities in close proximity. MEC finds
applications in homes, factories, and transport modes including trains and airplanes. However, thedesign of computation offloading policies for an MEC system, specifically, the decision of executing
a computation task at the mobile device or at the remote MEC server, should adapt to the network
randomness and uncertainties.
Now, Celimuge Wu at the University of Electro-Communications, Tokyo and colleagues in Finland,
USA, and China, report on the Deep-SARL, a double deep Q-network (DQN)-based online strategic
computation offloading algorithm to learn the optimal policy without knowing a priori knowledge ofnetwork dynamics (Fig. 1).
The computation offloading problem is modeled as a Markov decision process, where its objective isto maximize the long-term utility performance whereby an offloading decision is made based on thetask queue state, the energy queue state, and the channel qualities between mobile users and basestations. The researchers describe the adoption of a Q-function decomposition technique to enhance
the learning performance.
Numerical experiments based on TensorFlow show that their proposed learning algorithm achieves asignificant improvement in computation offloading performance compared with existing baselines,
showing an optimal tradeoff among the computation task execution delay, task drops, task queuing
delay, task failure penalty, and MEC service payment. Deep-SARL provides a novel and effective
approach to facilitate intelligence in edge computing under time-varying network dynamics.

The University of Electro-Communications 1-5-1 Chofugaoka, Chofu, Tokyo 182-8585© 2021 The University of Electro-Communications.

Vol.26, March 2021

Fig. 1 Deep-SARL-based strategic computation offloading in an MEC system.

Reference
1. Xianfu Chen, Honggang Zhang, Celimuge Wu, Shiwen Mao, Yusheng Ji, Mehdi Bennis, "Optimized
Computation Offloading Performance in Virtual Edge Computing Systems via Deep Reinforcement
Learning," IEEE Internet of Things Journal, Vol.6, no.3, pp. 4005-4018, June 2019.
DOI: 10.1109/JIOT.2018.2876279

The University of Electro-Communications 1-5-1 Chofugaoka, Chofu, Tokyo 182-8585© 2021 The University of Electro-Communications.

Vol.26, March 2021

Computing in close proximity: Edge intelligence with deep reinforcement learning

Tomonori Sadamoto
Assistant Professor
Department of Mechanical Engineering andIntelligent Systems, UEC Tokyo.
Assistant Professor Tomonori Sadamoto is an expertisein control theory, currently focusing on integrating control theory with reinforcement learning and powerengineering.
Reinforcement learning is a key methodology for controlling large-scale complex systems such as power grids andtransportation networks. However, the major contemporary learning theories currently used are unsuitable for realtime control because designers must repeat trials just for acquiring data. Instead, it is necessary to develop amethodology that is capable of real-time decision making.
With this background, Sadamoto and his colleagues have proposed a fast real-time learning method for large-scale
network systems. For example, it takes 140 seconds to learn when conventional methods are applied to 100dimensional level power systems. In contrast. Their method completes the task of learning in only 6 seconds— areduction of the learning time by 95% while maintaining control performance.

The central feature of their method is data compression, where redundant data is eliminated while minimizing
information loss by projecting data onto dominantly controllable subspace. Because of the smaller scale and lessinformation loss of the compressed data, the learning process can provide nearly optimal control with lower
computational complexity. Moreover, the applications of this approach can be extended to partly observablesystems where the availability of sensors is very limited.
These are new reinforcement learning methods, and in the language of control theory, can be interpreted as modelfree versions of optimal controller design through model order reduction.
Sadamoto is also working on the development of control-theoretic power engineering. The rapid spread ofrenewable energy resources has led to rapid and major changes in power grids and it is widely accepted thattraditional power engineering approaches are no longer applicable. So, system operators require innovative andsystematic methodology based on control theory.

The University of Electro-Communications 1-5-1 Chofugaoka, Chofu, Tokyo 182-8585© 2021 The University of Electro-Communications.

Vol.26, March 2021

For example, Sadamoto’s group has proposed a new wind turbine architecture to enhance the stability of windintegrated power systems. It is known that DFIG-type wind turbines can cause frequency oscillations in the grid.
Actions to control turbines, however, sometimes fail to dampen oscillations, and it is not clear when actions will fail.
In their research, Sadamoto and his colleagues theoretically showed that control failure is induced by parameterdependent uncontrollability of turbines. To solve this problem, they proposed adding a compensator to improve thecontrollability, thereby successfully enhancing the stability of wind-integrated power systems.
In related work, Sadamoto has proposed a new method for designing power flow schedules to enhance the stabilityof the entire grid. The stability of power systems is known to depend on the power flowing through them. However,current power flow design is based on economic optimality only. The proposed‘ method minimizes the H2 norm,
which is a measure of stability used in control theory, while ensuring economic optimality. This research is being
conducted in collaboration with Keio University in Tokyo.
These are examples of the innovative control-theoretic solutions being developed by Sadamoto and his colleagues
for the realization of next-generation power systems. In 2020 an article published by Sadamoto and his colleagues
received the IEEE Control Systems Magazine Outstanding Paper Award for bridging the gap between researchcommunities focused on control and power systems.
Further information
Research Highlight: Innovative automated control systems: Control-theoretic approach for fast online reinforcement
learning
Department website: http://www.sc.lab.uec.ac.jp/ts/index.html

The University of Electro-Communications 1-5-1 Chofugaoka, Chofu, Tokyo 182-8585© 2021 The University of Electro-Communications.

Vol.26, March 2021

UEC holds the 7th UEC Seminar in ASEAN, 2020 and the 2nd ASEAN - UEC Workshop

January 4, 2021
On November 21, 2020, the University of Electro-Communications (UEC) held the 7th UEC Seminar in ASEAN,
2020 and the 2nd ASEAN - UEC Workshop on Energy and AI online in collaboration with Bundung Institute ofTechnology (ITB), Indonesia, and the ECTI Association.
UEC Seminar in ASEAN has been annually organized by UEC ASEAN Research and Education Center (UAREC)
of UEC since 2015. Although the seminar and the workshop were held online this year due to the COVID-19
circumstances, they were co-hosted in Indonesia for the first time by Bandung Institute of Technology. The themeof this Seminar was "Collaboration implementation and Study at UEC", with the aim of further strengthening
collaboration based on the global network and promoting study abroad at UEC. Beginning with opening remarks by
Dr. I Gusti Bagus Baskara Nugraha of ITB and welcome remarks by UEC President Tano and Prof. Suhono
Supangkat of ITB, guest speeches were given by UEC's partner universities and government agencies. Following
the guest speeches, members of the new and continued partners of UEC gave speeches in anticipation of futureexchanges. Then, lectures on collaborative activities with industry and government agencies were given andinternational programs that provide opportunities to visit and study at UEC such as MICH program (the specialprogram of governmental scholarship), JUSST program (UEC's inbound student exchange program), and theSakura Science Plan, were introduced. The seminar was closed with closing remarks by Prof. Ishibashi, Director ofUAREC.
The 2nd ASEAN - UEC Workshop on Energy and AI was held jointly with the ECTI Association. The workshop
was started by the introductory talk from Prof. Kosin Chamnongthai, Vice Director of UAREC and former
President of ECTI Association. 10 Invited talks with a focus on energy and AI-related fields, which are widely
studied in Indonesia were delivered, and 24 poster presentations were given by mainly ASEAN and UEC students.
In the Award session, "Young Researcher Encouragement Award" was given to student poster presenters. Theworkshop was closed with closing remarks by Prof. Pham, the Vice Director of UAREC.
A record total of more than 150 participants gathered at the seminar and the workshop, and have deepened
exchanges for continued collaboration in ASEAN.
The next UEC Seminar and Workshop is planned to be held in Bangkok, Thailand, aiming for further international
research and education cooperation and industry-academia-government collaboration in the ASEAN region.

The University of Electro-Communications 1-5-1 Chofugaoka, Chofu, Tokyo 182-8585© 2021 The University of Electro-Communications.

Vol.26, March 2021

The University of Electro-Communications 1-5-1 Chofugaoka, Chofu, Tokyo 182-8585© 2021 The University of Electro-Communications.

Vol.26, March 2021

The University of Electro-Communications (UEC) in Tokyo
is a small, luminous university at the forefront of pure and applied sciences, engineering, andtechnology research. Its roots go back to the Technical Institute for Wireless Commutations,which was established in 1918 by the Wireless Association to train so-called wireless engineers
in maritime communications in response to the Titanic disaster in 1912. In 1949, the UEC wasestablished as a national university by the Japanese Ministry of Education, and moved in 1957
from Meguro to its current Chofu campus Tokyo.
With approximately 4,000 students and 350 faculty, UEC is regarded as a small university, butwith particular expertise in wireless communications, laser science, robotics, informatics, andmaterial science, to name just a few areas of research.
The UEC was selected for the Ministry of Education, Culture, Sports, Science and Technology (MEXT)
Program for Promoting the Enhancement of Research Universities as a result of its strengths inthree main areas: optics and photonics research, where we are number one for the numberof joint publications with foreign researchers; wireless communications, which reflects ourroots; and materials-based research, particularly on fuel cells.
International Public Relations
The University of Electro-Communications
1-5-1 Chofugaoka, Chofu, Tokyo 182-8585
E- mail : kokusai-k@office.uec.ac.jp
Website: http://www.uec.ac.jp/eng/

The University of Electro-Communications 1-5-1 Chofugaoka, Chofu, Tokyo 182-8585© 2021 The University of Electro-Communications.