Modifiye q-öğrenme yöntemi ile elektrik dağıtım sistemlerinin kendi kendini iyileştirme uygulaması
Self healing of electrical distribution systems by using modified q-learning method
Yazar: BEYZA NUR DİLBER
Danışman: PROF. DR. MUSTAFA BAĞRIYANIK
İstanbul Teknik Üniversitesi / Lisansüstü Eğitim Enstitüsü / Elektrik Mühendisliği Ana Bilim Dalı / Elektrik Mühendisliği Bilim Dalı
Derece: Yüksek Lisans, 2023
Konu: Elektrik ve Elektronik Mühendisliği = Electrical and Electronics Engineering

Özet:
Elektrik dağıtım şebekelerinin gün geçtikçe büyümesi ile birlikte; arıza durumlarında arızalı bölgenin en kısa sürede izole edilerek, şebekenin en efektif şekilde yeniden yapılandırılması ve buna bağlı olarak en uygun anahtarlama menavralarının gerçekleştirilmesi zorlaşmaktadır. Şebekenin yeniden yapılandırılmasını sağlarken, gerilim yük profili ve akım kısıtlarının limitler dahilde korunması, maksimum yük beslenecek şekilde menavra yapılması, hat kayıplarının minimumda tutulması, anahtarlarda yapılacak manevra sayılarının minimum olması, sistemin işletme yapısının ve radyalliğinin bozulmaması istenmektedir. Geleneksel elektrik dağıtım şebekelerinde, arıza durumunda gerçekleştirilen anahtarlama menavraları manuel olarak şebeke operatörleri tarafından gerçekleştirilmektedir. Bu da, operatörün şebeke yapısına hakim, deneyimli bir kişi olmasını gerektirir. Ayrıca anahtarlama manevra planının oluşturulması ve uygulanması zaman almaktadır, her zaman çok kolay olmamaktadır.
Akıllı şebekeler kapsamında, dağıtım otomasyonu uygulamaları ile birlikte elektrik dağıtım şebekelerinin efektif çalışmasına yönelik birçok uygulama gerçekleştirilmektedir. Kendi kendini iyileştirme uygulaması da bunlardan biridir. Arıza veya kesinti durumlarında, hatta optimum kontrol sağlanarak en kısa sürede optimum anahtarlama manevraları gerçekleştirilir. Dağıtım sisteminde bulunan normalde açık konumda bulunan bağlantı anahtarları ve normalde kapalı konumda bulunan ayırıcı anahtarlar üzerinde gerçekleştirilecek manevralar ile uygun yük aktarımı yapılarak arızasız bölgelerin enerjilendirilmesi sağlanır. 
Bu çalışmada, arıza durumunda dağıtım sisteminin kendi kendini iyileştirmesine yönelik bir yaklaşım sunulmuştur. Literatürde dağıtım sisteminde kendi kendini iyileştirmeye yönelik yapılan çalışmalar incelenmiştir. Çalışma kapsamında 16 baralı radyal bir örnek sistem olası arıza senaryolarına karşın, Makine Öğrenmesinin bir dalı olan Takviyeli Öğrenme Metoduna ait Q-Öğrenme algoritmasının modifiye hali kullanılarak kendi kendini iyileştirme uygulaması gerçekleştirilmiştir. Uygulama, manevra planlamasında gerilim ve akım kısıtlarını göz önünde bulundururken; ayrıca radyalliğin korunması, sistemdeki tüm yüklerin beslenmesi şartını da içermektedir. Bu kısıtlara bağlı olarak; hat kaybının ve anahtarlama manevralarının minimumda tutulması amaç fonksiyonuna dahil edilmiştir. Çalışma kapsamında, hat kaybının anahtarlama kararındaki önem katsayısı, manevra sayısının katsayısından yüksek tutulmuştur. Bu katsayılar, dağıtım sisteminin yapısına ve dağıtım sistemindeki maliyet analizlerine bağlı olarak değişebilir. Çalışmada, yük akışı analizleri için MATPOWER, algoritmanın tamamı için MATLAB programları kullanılmıştır. Örnek dağıtım sistemi üzerindeki olası arıza durumları incelenmiş, algoritmanın ödül matrisinden elde ettiği optimum manevralar elde edilmiştir.

Abstract:
With the increasing complexity of electricity distribution networks day by day; in case of faults, it becomes difficult to isolate the faulty area as soon as possible, to reconfigure the network in the most efficient way and to perform the most suitable switching maneuvers. It is preferred to maintain the voltage load profile and current constraints within the allowed limits while reconfiguring the network, to move in such a way as to feed the maximum amount of load, to minimise line losses, to reduce the number of switching manoeuvres required, and to avoid breaking the system's radiality and operating structure.
The manual switching operations used in conventional electrical distribution networks are carried out by the network operators. This requires the operator to be a quite experienced person with a good command of the network structure. Additionally, the switching manoeuvre plan's preparation and implementation might take long time, and might be not always easy due to several conditions. Therefore, it affects flexibility of electrical distribution systems which is essential. 
The quality, efficiency, reliability, stability and flexibility of electricity distribution networks that supply energy to consumers are crucial. Studies have been being carried out to minimize the duration and number of outages, losses and voltage drops in energy, and to reduce costs. With the increase in the number of equipment in the network and the growth of the networks, the coordination between the equipments should be done in a planned and more flexible way. Otherwise, the number of outages and downtime might increase, load imbalances might occur. Accordingly, developments within the scope of Smart Grids and Distribution Automation, which provides the modernization of electricity networks, are increasing day by day. The main purpose of these applications is to increase the flexibility, reliability, efficiency, sustainability, quality and reliability of the network. Although the cost of these studies is high in the first stage; with the increase and spread of production over time, Smart Grid costs are expected to decrease considerably. Various applications are carried out by Distribution Automation for the flexibility of electricity distribution networks, thus improving the reliability and quality parameters of the network.
Within the scope of Smart Grids, many applications are carried out for the effective operation of electricity distribution networks with distribution automation applications. Self healing of distribution networks is one of the applications of distribution automation. With self healing applications, in the case of a fault or outage, the quickest switching manoeuvres are carried out by providing the optimum control on the line. The suitable load transfer is made to reenergise the fault-free areas through the actions to be taken on the tie switches which are normally opened and sectionalizing switches which are normally closed. 
In this study, an approach for self-healing of the distribution system in case of fault has been presented. The literature has been reviewed for studies on self-healing in the distribution system as the methods used and the applications applied. The studies examined include applications such as fault location detection, reconfiguration of the system for different purposes regardless the fault event, and restoration of the system in case of fault conditions. Generally, heuristic optimization methods have been used for self healing application. However, heuristic optimization techniques might not always provide the best solution because most of them are random search-based algorithms. Therefore, these methods do not always give the same results in the switching decision, and the processing time, which is critical for the restoration process of distribution systems, might take quite a long time. For this reason, intelligent methods such as Machine Learning have recently started to be applied within the scope of self-healing. 
Within the scope of this study, a radial sample system with 16 busbars, against possible fault scenarios, a self-healing application has been carried out by using the modified version of the Q-Learning algorithm of the Reinforced Learning Method, which is a branch of Machine Learning. Q-Learning algorithm is based on action-reward policy. Main parameters of Q-Learning algorithm are environment, state, action and reward. Environment is the distribution system in this study. States are switching positions in faulty conditions. Actions include possible switching operations in case of faults. Actions in this study have been created considering the constraints (voltage, current, radiality, feeding all loads in the system). Reward table have been crated according to line loss and number of maneuvers.
While the application considers voltage and current constraints in maneuver planning; it also includes the condition of maintaining radiality, feeding all loads in the system. The significance coefficient of the line loss in the switching decision has been kept greater than the coefficient of the number of manoeuvres within the parameters of the application. This demonstrates that it is preferable to maintain the loss in the line and the number of manoeuvres in the switching selection at a minimum. Line loss, however, plays a bigger role in switching decision-making. These coefficients may vary depending on the structure of the distribution system and the cost analysis in the distribution system.
In the study, MATPOWER has been used for load flow analysis and MATLAB program has been used for the whole algorithm. MATPOWER is running in MATLAB program. Possible fault situations on the sample distribution system have been examined, and optimum maneuvers obtained by the modified Q-Learning algorithm from the reward matrix has been obtained.
In the first part of the thesis, the introduction and literature review have been presented, the purpose of the thesis has been explained. 
In the second part, general information about the structures of electricity distribution systems including different network topologies, flexibility of distribution networks has been given.
Distribution automation has been addressed in the third chapter including main principle of those applications, and examples of some applications.
The fourth chapter discusses in greater detail about one of the applications for distribution automation—the self-healing application of the distribution system. Working principle of the networks with self healing, impacts of self healing applications on distribution networks, objective functions and constraints have been mentioned in this chapter.
The fifth chapter covers Machine Learning and the Q-Learning algorithm. Basic principle of Machine Learning has been explained first. Then, Reinforcement Learning, one of the methods of Machine Leaning, has been explained. After that, Q-Learning information has been given with more details.
In section six, the proposed method's approach, modified Q-Learning, to the example system has been detailed. Fault scenarios have been assessed with the proposed method.
Finally, the conclusion of the thesis and an overview of future work have been presented.