Intelligent Resource Allocation for Massive Internet of Things Traffic
Creators: name: Arruda, Carlos E.
Publisher: name: Zenodo
Year: 2022
Subjects: IoT
Intelligent Traffic Orchestration
Reinforcement Learning
SARSA,
Massive Traffic
Dynamic Traffic
PSIoT Framework
Smart Grid

description: The Internet of Things (IoT) is considered a major trend in computing and in specific areas such as Smart Cities, Smart Grid, Industry 4.0, and mobile applications based on 5G. Typically, this set of technologies requires the orchestration of heterogeneous resources that are allocated over distinct infrastructures such as Cloud Computing, Cloud of Things, Datacenters, and network backbones. Consistent with this demand, the PSIoT-Orch framework was designed to orchestrate massive IoT traffic and to allocate network resources between Aggregators and Consumers in a Publish / Subscribe strategy. This dissertation aims to build an intelligent module for PSIoT-Orch that is capable of handling data types with different transmission requirements, aiming at the efficient use of a limited communication link. The proposed component uses Reinforcement Learning, more specifically, the SARSA algorithm to dynamically adjust the available bandwidth according to transmission priority. This solution, named PSIoT-SARSA, is validated in a simulation environment under the statistical methods of Analysis of Variance and Response Surface Analysis and, at the end of the study, it is observed that it obtained promising results. The contributions are focused on gathering an approach that allows allocating bandwidth in an intelligent way, allowing efficient scheduling of the IoT flow, in the scenario of the Smart Grid.
description: {"references": ["AL-KHATIB, O.; HARDJAWANA, W.; VUCETIC, B. Queuing analysis for smart grid communications in wireless access networks. In: 2014 IEEE International Conference on Smart Grid Communications (SmartGridComm). [S.l.: s.n.], 2014. p. 374\u2013379.", "ALAA, Y. et al. Lte dynamic scheduling scheme for massive m2m and h2h communication. In: 2017 IEEE 8th Annual Ubiquitous Computing, Electronics and Mobile Communication Conference (UEMCON). [S.l.: s.n.], 2017. p. 478\u2013482.", "ANDERSON-COOK, C. Design of experiments: a single experiment or sequential learning. Quality Progress, American Society for Quality, v. 40, n. 3, p. 71, 2007.", "AYOUBI, S. et al. Machine learning for cognitive network management. IEEE Communications Magazine, v. 56, n. 1, p. 158\u2013165, Jan 2018. ISSN 0163-6804.", "BALLA, H. A.; SHENG, C. G.; WEIPENG, J. Reliability enhancement in cloud computing via optimized job scheduling implementing reinforcement learning algorithm and queuing theory. In: IEEE. Data Intelligence and Security (ICDIS), 2018 1st International Conference on. [S.l.], 2018. p. 127\u2013130.", "BINDER, R. V. Optimal scheduling for combinatorial software testing and design of experiments. In: 2018 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). [S.l.: s.n.], 2018. p. 295\u2013301.", "BORGIA, E. The internet of things vision: Key features, applications and open issues. Computer Communications, Elsevier, v. 54, p. 1\u201331, 2014.", "BURNSIDE, S. The application of design of experiments to rf systems. In: IEEE. Applied Computational Electromagnetics Society Symposium (ACES), 2018 International. [S.l.], 2018. p. 1\u20132.", "CAMARGO, L. F. R.; MOREIRA, V.; VACCARO, G. L. R. Aplica\u00e7\u00e3o das t\u00e9cnicas de planejamento e an\u00e1lise de experimentos no desenvolvimento de novos produtos em uma empresa de saneantes. Estudos Tecnol\u00f3gicos, v. 5, n. 3, p. 404\u2013420, 2009. ISSN 1808-7310. Dispon\u00edvel em: . Acesso em: 2 jan. 2019.", "COLAK, I. et al. A survey on the critical issues in smart grid technologies. Renewable and Sustainable Energy Reviews, Elsevier, v. 54, p. 396\u2013405, 2016.", "CORAZZA, M.; SANGALLI, A. Q-learning and sarsa: a comparison between two intelligent stochastic control approaches for financial trading. University Ca'Foscari of Venice, Dept. of Economics Research Paper Series No, v. 15, 2015. Dispon\u00edvel em: . Acesso em: 1o maio 2019.", "FAUSTINI, P. H. A. et al. Emprego de nfv e aprendizagem por refor\u00e7o para detectar e mitigar anomalias em redes definidas por software. Anais do Simp\u00f3sio Brasileiro de Redes de Computadores e Sistemas Distribu\u00eddos (SBRC), v. 35, n. 1/2017, 2017. ISSN 2177-9384. Dispon\u00edvel em: . Acesso em: 5 maio 2018.", "FERREIRA, J. C.; PATINO, C. M. O que realmente significa o valor-p? Jornal Brasileiro de Pneumologia, v. 41, n. 5, p. 485, 2015. Dispon\u00edvel em: . Acesso em: 5 nov. 2018.", "FREITAS, L. A. S.; MATOS, F. B.; NOGUEIRA, P. E. Estudo experimental sobre paralelismo na linguagem go usando goroutines. In: 13o Encontro Anual de Computa\u00e7\u00e3o (EnAComp). Goi\u00e2nia: UFG, 2017. Dispon\u00edvel em: . Acesso em: 16 jan. 2019.", "HAPP, D.; WOLISZ, A. Limitations of the pub/sub pattern for cloud based iot and their implications. In: 2016 Cloudification of the Internet of Things (CIoT). [S.l.: s.n.], 2016. p. 1\u20136.", "HENNING, E. et al. Rstudio como suporte no ensino de planejamento de experimentos. In: 41o Congresso Brasileiro de Educa\u00e7\u00e3o em Engenharia (COBENGE). Gramado: [s.n.], 2013. Dispon\u00edvel em: . Acesso em: 27 dez. 2018.", "HOANG, D. T. et al. Optimal data scheduling and admission control for backscatter sensor networks. IEEE Transactions on Communications, v. 65, n. 5, p. 2062\u20132077, Maio 2017. ISSN 0090-6778.", "JANG, H.-C.; HUANG, C.-W.; YEH, F.-K. Design a bandwidth allocation framework for sdn based smart home. In: IEEE. Information Technology, Electronics and Mobile Communication Conference (IEMCON), 2016 IEEE 7th Annual. [S.l.], 2016. p. 1\u20136.", "JI, X.; KANG, S. A study on problem solving strategy using experiment of design. In: 2017 China Semiconductor Technology International Conference (CSTIC). [S.l.: s.n.], 2017. p. 1\u20136.", "JIANG, T. Power monitoring electronic/multimedia traffic scheduling in cognitive radio based smart grid. In: 2016 International Conference on Smart Grid and Clean Energy Technologies (ICSGCE). [S.l.: s.n.], 2016. p. 80\u201383.", "JIANG, Z.-Y. et al. Optimal dynamic bandwidth allocation for complex networks. Physica A: Statistical Mechanics and its Applications, Elsevier, v. 392, n. 5, p. 1256\u20131262, 2013.", "KABALCI, Y. A survey on smart metering and smart grid communication. Renewable and Sustainable Energy Reviews, Elsevier, v. 57, p. 302\u2013318, 2016.", "KARIMI, B.; NAMBOODIRI, V.; JADLIWALA, M. On the scalable collection of metering data in smart grids through message concatenation. In: 2013 IEEE International Conference on Smart Grid Communications (SmartGridComm). [S.l.: s.n.], 2013. p. 318\u2013323.", "KARUPONGSIRI, C. et al. A novel scheduling technique for smart grid data on lte networks. In: 2013, 7th International Conference on Signal Processing and Communication Systems (ICSPCS). [S.l.: s.n.], 2013. p. 1\u20135.", "KHALIFA, T. et al. Transport protocol for smart grid infrastructure. In: 2010 Second International Conference on Ubiquitous and Future Networks (ICUFN). [S.l.: s.n.], 2010. p. 320\u2013325. ISSN 2165-8528.", "KHAN, A. A.; REHMANI, M. H.; REISSLEIN, M. Requirements, design challenges, and review of routing and mac protocols for cr-based smart grid systems. IEEE Communications Magazine, v. 55, n. 5, p. 206\u2013215, May 2017. ISSN 0163-6804.", "KIUMARSI, B. et al. Optimal and autonomous control using reinforcement learning: A survey. IEEE transactions on neural networks and learning systems, IEEE, v. 29, n. 6, p. 2042\u20132062, 2018.", "KLEIN, D.; ABBEEL, P. Artificial intelligence: Markov decision processes. 2013. Dispon\u00edvel em: . Acesso em: 30 abr. 2019.", "KLEIN, D.; ABBEEL, P. Artificial intelligence: Markov decision processes ii. 2013. Dispon\u00edvel em: . Acesso em: 1o maio 2019.", "KLEIN, D.; ABBEEL, P. Artificial intelligence: Reinforcement learning. 2013. Dispon\u00edvel em: . Acesso em: 2 maio 2019.", "KLEIN, D.; ABBEEL, P. Artificial intelligence: Reinforcement learning ii. 2013. Dispon\u00edvel em: . Acesso em: 2 maio 2019.", "LAW, A. M. A tutorial on design of experiments for simulation modeling. In: IEEE. Simulation Conference (WSC), 2017 Winter. [S.l.], 2017. p. 550\u2013564.", "LI, C. et al. Urban traffic signal learning control using sarsa algorithm based on adaptive rbf network. In: 2009 International Conference on Measuring Technology and Mechatronics Automation. [S.l.: s.n.], 2009. v. 3, p. 658\u2013661. ISSN 2157-1473.", "LILITH, N.; DOGANCAY, K. Distributed dynamic call admission control and channel allocation using sarsa. In: 2005 Asia-Pacific Conference on Communications. [S.l.: s.n.], 2005. p. 376\u2013380. ISSN 2163-0771.", "LIU, N. et al. A key management scheme for secure communications of advanced metering infrastructure in smart grid. IEEE Transactions on Industrial Electronics, v. 60, n. 10, p. 4746\u20134756, Oct 2013. ISSN 0278-0046.", "MARTINS, J. S. B. et al. Uma Vis\u00e3o Tutorial dos Modelos de Aloca\u00e7\u00e3o de Banda como Mecanismo de Provisionamento de Recursos em Redes IP/MPLS. Revista de Sistemas e Computa\u00e7\u00e3o, v. 5, n. 2, p. 144\u2013155, dez. 2015. ISSN 2237-2903.", "MATUCK, P. J. Uso de aprendizado por refor\u00e7o para controle din\u00e2mico de tr\u00e1fego urbano. Disserta\u00e7\u00e3o (Mestrado em Ci\u00eancia e Tecnologia da Computa\u00e7\u00e3o) \u2014 Universidade Federal de Itajub\u00e1, Itajub\u00e1, Abril 2018. Dispon\u00edvel em: . Acesso em: 2 nov. 2018.", "MEER, A. A. van der et al. Design of experiments aided holistic testing of cyber-physical energy systems. In: IEEE. 2018 Workshop on Modeling and Simulation of Cyber-Physical Energy Systems (MSCPES). [S.l.], 2018. p. 1\u20137.", "MIJUMBI, R. et al. Design and evaluation of learning algorithms for dynamic resource management in virtual networks. In: 2014 IEEE Network Operations and Management Symposium (NOMS). [S.l.: s.n.], 2014. p. 1\u20139. ISSN 1542-1201.", "MIRANDA, R. de C.; PINHO, A. F. de; MONTEVECHI, J. A. B. An\u00e1lise de signific\u00e2ncia dos par\u00e2metros de um algoritmo gen\u00e9tico atrav\u00e9s do planejamento de experimentos. CADERNOS DO IME \u2013 S\u00e9rie Estat\u00edstica, UERJ, v. 31, p. 1\u201314, 2011. ISSN 1413-9022. Dispon\u00edvel em: . Acesso em: 20 jan. 2019.", "MONTEIRO, S. T.; RIBEIRO, C. H. Desempenho de algoritmos de aprendizagem por refor\u00e7o sob condi\u00e7\u00f5es de ambiguidade sensorial em rob\u00f3tica m\u00f3vel. Sba: Controle &amp; Automa\u00e7\u00e3o Sociedade Brasileira de Automatica, SciELO Brasil, v. 15, n. 3, p. 320\u2013338, 2004.", "MONTGOMERY, D. C. Introdu\u00e7\u00e3o Ao Controle Estat\u00edstico Da Qualidade . 4. ed. Rio de Janeiro: LTC, 2004. 365\u2013422 p.", "MONTGOMERY, D. C. Design and analysis of experiments. 8. ed. [S.l.]: John wiley &amp; Sons, 2013.", "MORAES, P.; REALE, R. F.; MARTINS, J. S. B. A Publish/Subscribe QoS-aware Framework for Massive IoT Traffic Orchestration. In: 6th International Workshop on ADVANCEs in ICT Infrastructures and Services (ADVANCE 2018). Santiago, Chile: [s.n.], 2018. p. 1\u201314. Dispon\u00edvel em: . Acesso em: 12 mar. 2018.", "MYERS, R. H.; MONTGOMERY, D. C.; ANDERSON-COOK, C. M. Response surface methodology: process and product optimization using designed experiments. [S.l.]: John Wiley &amp; Sons, 2016.", "NODE.JS. Node.js is a javascript runtime built on chrome's v8 javascript engine. 2018. Dispon\u00edvel em: . Acesso em: 12 mar. 2018.", "OLIVEIRA, E. M.; REALE, R. F.; MARTINS, J. S. B. Cognitive management of bandwidth allocation models with case-based reasoning - evidences towards dynamic bam reconfiguration. In: 2018 IEEE Symposium on Computers and Communications (ISCC). [S.l.: s.n.], 2018. p. 00397\u201300403. ISSN 1530-1346.", "OTTONI, A. L. C.; NEPOMUCENO, E. G.; OLIVEIRA, M. S. de. An\u00e1lise da defini\u00e7\u00e3o de par\u00e2metros do aprendizado por refor\u00e7o no desempenho de um manipulador rob\u00f3tico. For Science - Revista Cient\u00edfica do IFMG, v. 5, n. 3, 2017. ISSN 2318-6356. Dispon\u00edvel em: . Acesso em: 12 mar. 2018.", "PANCHISHIN. Sarsa. 2018. Dispon\u00edvel em: . Acesso em: 14 abr. 2018.", "PELLEGRINI, J.; WAINER, J. Processos de decis\u00e3o de markov: um tutorial. Revista de Inform\u00e1tica Te\u00f3rica e Aplicada, v. 14, n. 2, p. 133\u2013179, 2007. Dispon\u00edvel em: . Acesso em: 30 abr. 2019.", "PIMENTA, C. D. et al. Planejamento de experimentos (doe) aplicado no processo de t\u00eampera e revenimento de arames de a\u00e7o sae 9154. J\u00e2nus, v. 4, n. 5, 2009. Dispon\u00edvel em: . Acesso em: 27 dez. 2018.", "PORTAL-ACTION. 3.1.1 - experimentos fatoriais 22. 2019a. Dispon\u00edvel em: . Acesso em: 19 jan. 2019.", "PORTAL-ACTION. 6.1 - experimentos para superf\u00edcie de resposta de segunda ordem. 2019b. Dispon\u00edvel em: . Acesso em: 23 jan. 2019.", "PRAUCHNER, D. C. K.; MARTINS, R. S. de M.; PADOIN, E. L. Aplica\u00e7\u00e3o do algoritmo sarsa \u00e0 coleta de lixo: avalia\u00e7\u00e3o de par\u00e2metros. Universidade Regional do Noroeste do Estado do Rio Grande do Sul, Iju\u00ed, 2014. Dispon\u00edvel em: . Acesso em: 2 nov. 2018.", "R. The r project for statistical computing. 2018. Dispon\u00edvel em: . Acesso em: 12 mar. 2018.", "RACHKIDI, E. Modelling and placement optimization of compound services in a converged infrastructure of cloud computing and internet of things. Tese (Doutorado) \u2014 Universit\u00e9 Paris-Saclay; Universit\u00e9 d'Evry-Val-d'Essonne, 2017. Dispon\u00edvel em: . Acesso em: 30 mar. 2018.", "REALE, W. d. C. P. N. R. F.; MARTINS, J. S. B. Routing in ds-te networks with an opportunistic bandwidth allocation model. In: IEEE COMPUTER SOCIETY. Proceedings of the 2012 IEEE Symposium on Computers and Communications (ISCC). [S.l.], 2012. p. 88\u201393.", "REHMANI, M. H. et al. Software defined networks based smart grid communication: A comprehensive survey. arXiv preprint arXiv:1801.04613, 2018. Dispon\u00edvel em: . Acesso em: 30 mar. 2018.", "ROCHA, K. R. da; J\u00daNIOR, A. J. B. Anova medidas repetidas e seus pressupostos: An\u00e1lise passo a passo de um experimento. Revista Eletr\u00f4nica Perspectivas da Ci\u00eancia e Tecnologia, v. 10, p. 29, 2018. ISSN 1984-5693. Dispon\u00edvel em: . Acesso em: 26 dez. 2018.", "RSTUDIO. Rstudio software. 2018. Dispon\u00edvel em: . Acesso em: 12 mar. 2018.", "SARMIENTO, M. G. C. Aplica\u00e7\u00e3o da an\u00e1lise de cumeeira para a otimiza\u00e7\u00e3o de processos, modelados por superf\u00edcies de resposta, com experimentos planejados. Disserta\u00e7\u00e3o (Mestrado) \u2014 Pontif\u00edcia Universidade Cat\u00f3lica do Rio de Janeiro, Rio de Janeiro, Abril 2014. Dispon\u00edvel em: . Acesso em: 23 jan. 2019.", "SHAO, S. et al. Traffic scheduling for wireless meter data collection in smart grid communication network. In: Fifth International Conference on Computing, Communications and Networking Technologies (ICCCNT). [S.l.: s.n.], 2014. p. 1\u20137.", "SILVA, T. S. J. da. Aplica\u00e7\u00e3o do planejamento experimental (doe) no processo de desenvolvimento de fitoter\u00e1picos na ind\u00fastria farmac\u00eautica. Revista Especialize On-line IPOG, Goi\u00e2nia, v. 1, n. 10, 2015. ISSN 2179-5568. Dispon\u00edvel em: . Acesso em: 27 dez. 2018.", "SIQUEIRA, D. et al. Uma an\u00e1lise de desempenho sobre uma microfog. Workshop em Desempenho de Sistemas Computacionais e de Comunica\u00e7\u00e3o (WPerformance_ CSBC), v. 17, n. 1/2018, 2018. ISSN 2595-6167. Dispon\u00edvel em: . Acesso em: 16 jan. 2019.", "SONGXI, L. et al. Traffic scheduling with sustainable cyber physical systems applying in smart grid. In: 2016 Seventh International Green and Sustainable Computing Conference (IGSC). [S.l.: s.n.], 2016. p. 1\u20136.", "SRINIVASAN, S. M.; TRUONG-HUU, T.; GURUSAMY, M. Deadline-aware scheduling and flexible bandwidth allocation for big-data transfers. IEEE Access, IEEE, v. 6, p. 74400\u201374415, 2018.", "SUTTON, R. S.; BARTO, A. G. Reinforcement learning: An introduction. IEEE Transactions on Neural Networks, v. 9, n. 5, Sept 1998. ISSN 1045-9227.", "TANABE, I. Development of a tool for the easy determination of control factor interaction in the design of experiments and the taguchi methods. In: 2017 International Conference on Control, Artificial Intelligence, Robotics Optimization (ICCAIRO). [S.l.: s.n.], 2017. p. 301\u2013306.", "TAO, Y.; WEI, H.; YIXIN, Z. A dynamic allocation strategy of bandwidth of networked control systems with bandwidth constraints. In: IEEE. Advanced Information Management, Communicates, Electronic and Automation Control Conference (IMCEC), 2016 IEEE. [S.l.], 2016. p. 1153\u20131158.", "TERAN-SOMOHANO, A.; SMITH, A. E.; YILMAZ, L. Model alignment using optimization and design of experiments. In: 2017 Winter Simulation Conference (WSC). [S.l.: s.n.], 2017. p. 1288\u20131299. ISSN 1558-4305.", "TESAURO, G. Reinforcement learning in autonomic computing: A manifesto and case studies. IEEE Internet Computing, v. 11, n. 1, p. 22\u201330, Jan 2007. ISSN 1089-7801.", "TUUMOV\u00c1, O.; KUPKA, L.; NETOLICKY` , P. Design of experiments approach and its application in the evaluation of experiments. In: IEEE. 2018 International Conference on Diagnostics in Electrical Engineering (Diagnostika). [S.l.], 2018. p. 1\u20134.", "VSCODE. Visual studio code - code editing. 2018. Dispon\u00edvel em: . Acesso em: 12 mar. 2018.", "WAINER, J. et al. M\u00e9todos de pesquisa quantitativa e qualitativa para a ci\u00eancia da computa\u00e7\u00e3o. Atualiza\u00e7\u00e3o em inform\u00e1tica, Sociedade Brasileira de Computa\u00e7\u00e3o e Editora PUC-Rio, v. 1, p. 221\u2013262, 2007. Dispon\u00edvel em: . Acesso em: 16 jan. 2019.", "YAGHMAEE, M. H. et al. Quality of service guarantee in smart grid infrastructure communication using traffic classification. In: 22nd International Conference and Exhibition on Electricity Distribution (CIRED 2013). [S.l.: s.n.], 2013. p. 1\u20134.", "ZHOU, Y. et al. A multi-applications comprehensive traffic prediction model for the electric power data network. In: 2016 18th Asia-Pacific Network Operations and Management Symposium (APNOMS). [S.l.: s.n.], 2016. p. 1\u20134.", "ZIPPERER, A. et al. Electric energy management in the smart home: Perspectives on enabling technologies and consumer behavior. Proceedings of the IEEE, v. 101, n. 11, p. 2397\u20132408, Nov 2013. ISSN 0018-9219."]}