—In order to make full use of heterogeneous hard- ware, it is necessary to have a technical skill of hardware such as OpenCL, and the current situation is that the barrier is high. Based on this background, I have proposed environment- adaptive software that enables high-performance operation by automatically converting application code written for normal CPUs by engineers according to the deployed environment and setting appropriate amount of resources. Until now, I only considered conversions and settings before the start of operation. In this paper, I verify that the logic is reconﬁgured according to the usage characteristics during operation. I conﬁrm that the application running on the FPGA is reconﬁgured into another application according to the usage characteristics. Index Terms —Environment Adaptive Software, Automatic Of- ﬂoading, FPGA, Reconﬁguration during Operation, Cost Perfor- mance. I. INTRODUCTION From the deceleration prediction of Moore’s Law, a multi- core CPU, GPU (Graphics Processing Unit) or FPGA (Field Programmable Gate Array) increases, which is not only in- creasing the degree of semiconductor integration and the number of clocks of one core CPU (Central Processing Unit) . This type of heterogeneous hardware has come to be used for normal application operation. Microsoft is making efforts such as searching with FPGA [1], and Amazon provides FPGA and GPU instances [2] using cloud technologies ([3]-[8]). In addition, the use of small devices such as IoT devices is increasing (e.g., [9]-[18] as heterogeneous hardware. However, in order to efﬁciently use heterogeneous hardware that is not a single core CPU, it is necessary to create and set programs according to the hardware speciﬁcation, which is a high barrier for most engineers. High knowledge and skill of OpenMP (Open Multi-Processing) [19] for multi-core CPU, CUDA (Compute Uniﬁed Device Architecture) [20] for GPU to achieve GPGPU (General Purpose GPU) [21], OpenCL (Open Computing Language) [22] for FPGA, assembly for IoT devices, or so on are often required. Even if we use high level description techniques such as [23]-[25], performance improvement needs much skills. In order to increase the use of heterogeneous hardware, we think that a platform that enables even ordinary engineers without high knowledge to make the best use of them is nec- essary. A platform analyzes software that describes processing with the same logic as a normal CPU, appropriately convertsand sets it according to the environment of the deployment destination (multi-core CPU, GPU, FPGA, etc.), and adapts to the environment. In the future, these platforms will be required to perform the adaption. Therefore, I have proposed environment-adaptive software which automatically converts, sets resources, determines the placement and others of the program code once written for normal CPU so that the GPU, FPGA, multi-core CPU that exist in the environment of the placement destination can be used, and makes the applications high performances. At the same time, as elements of environment-adaptive software, I have proposed and evaluated a method of automatically ofﬂoading loop statements and functional blocks of codes to GPUs and FPGAs, and a method of appropriately assigning the amount of processing resources such as GPUs [26]-[31]. However, my environmental adaptation has been based on the premise that adaptation processing such as conversion is performed before the start of operation of the application, and it was not studied that it will be reconﬁgured according to changes in usage characteristics after the start of operation. This paper focuses on the reconﬁguration of FPGA logic, while reconﬁguring software according to the usage char- acteristics during operation. Except for special applications such as the circuit reconﬁguration of artiﬁcial satellite while using FPGA for accelerating normal applications, there is no example in the commercial cloud, which reconﬁgures FPGA logic according to usage characteristics during application operation. FPGA reconﬁguration during application operation is difﬁcult and effective, we think. First, we ofﬂoad a normal CPU program to the FPGA and start operation, analyze the request characteristics, propose to change the FPGA logic to another program, and examine and evaluate a method for reconﬁguration with less user effect. The effectiveness of the proposed method is conﬁrmed through the FPGA conﬁgura- tion of the existing application and the reconﬁguration during operation. II. FPGA RECONFIGURATION DURING OPERATION A. Review of the automatic FPGA ofﬂoad method before operation start I review the automatic FPGA ofﬂoad method of the loop statement that has been veriﬁed by my previous papers [31] simply. Fig. 1. Automatic FPGA ofﬂoad of loop statements For the automatic FPGA ofﬂoad of the loop statement is made some ofﬂoad patterns and exploring high speed patterns through Clang [32] analysis and veriﬁcation environment mea- surement by focusing on loop statements with high arithmetic intensity and loops numbers (See, Fig. 1). At the time of the GPU, the combination was performed in most loop statements by Genetic Algorithms [33], and the measurement of 1000 times scale was performed to explore the optimal pattern. In FPGA, it takes more than 6 hours to compile, so the number of times measurements is narrowed down. B. Basic policy for FPGA reconﬁguration during operation By II.A method, the application speciﬁed by the user can automatically ofﬂoad the loop statements suitable for FPGA to FPGA. After ofﬂoad to the production environment used by the user, the user can check the actual performance and price in the production environment, and the user will start or not using the application. However, the performance optimization test case used in II.A (the item to be measured when comparing the performance in multiple ofﬂoad patterns) uses the assumed usage data speciﬁed by the user before the operation starts. It is possible that it will be greatly separated from the data that will be used after the operation starts. Therefore, in II.B, the usage after the operation starts is different from the initial assumption, and the performance may be improved by ofﬂoading other logic to FPGA. At this time, it is considered to be reconﬁgured with a low user impact of FPGA logic. The reconﬁguration may be changed to a different loop statement ofﬂoad in the same application, or may be changed to ofﬂoad of different applications. There are two types of FPGA reconﬁguration: dynamic re- conﬁguration and static reconﬁguration. The former is a tech- nology that changes the circuit conﬁguration while running the FPGA and the time for rewriting is msec order, the latter is a technology that stops the FPGA and then changes the circuit conﬁguration, and the break time is about 1 second. Depending on the degree of user impact of the break time, we can select areconﬁguration method provided by FPGA vendors. However, either method has a break time, and it requires a test for operation conﬁrmation after reconﬁguration, therefore, I do not think that it should be reconﬁgured frequently. To restrict frequent reconﬁguration, reconﬁgurations are only proposed when the improvement effect is higher than the threshold. The reconﬁguration processing begins with an analysis of request tendency for a certain period such as one month. The functions of the proposed method analyze the request tendency and understand whether there are applications which load are higher or the same than that of the currently ofﬂoad application. Next, applications with high load are executed in the veriﬁcation environment for the FPGA ofﬂoad opti- mization using data actually used in production use instead of the assumed usage data. Here, it is determined whether the new ofﬂoad pattern found by veriﬁcation has a much higher improvement effect than the current ofﬂoad pattern or less comparing the threshold value. If the improvement exceeds the threshold, a reconﬁguration is proposed to the user. After the user acceptance, the production environment is reconﬁgured. At the time of reconﬁguration, it is reconﬁgured to reduce user impact as much as possible. C. Method proposal of FPGA reconﬁguration during opera- tion Based on the basic policy of II.B, II.C proposes a concrete reconﬁguration method. The reconﬁguration method consists of 6 steps, and each step is explained in detail. In particular, step 1 is complicated, so I will add supplementary explanation it at the end. 1. For a certain period of time (long term), production request data is analyzed, and multiple applications with high processing time load are identiﬁed and production representa- tive data when using the applications are acquired. 1-1. The actual processing time and the number of usage times from each application usage history for a certain period are calculated. However, for an application that is ofﬂoaded to FPGA, the processing time is calculated assuming that it is not ofﬂoaded. From the test history in the assumed usage data before the start of operation, (actual processing time with CPU processing only)/(actual processing time with FPGA ofﬂoad) is calculated to set the improvement coefﬁcient. Next, the total processing time is used for comparing to calculate the sum value of (the improvement coefﬁcient)*(the actual processing time). 1-2. The total actual processing time with all applications are compared. 1-3. Based on the total actual processing time with all ap- plications, multiple applications with high load are identiﬁed. 1-4. Request data for a certain period (short term) of the high load applications are obtained. The distributions of request data according to the data size are created. 1-5. From the actual request data corresponding to the Mode value of the data size distribution, one data is selected as the production representative data. 2. Ofﬂoad patterns are extracted through veriﬁcation envi- ronment measurement, which speeds up the test case of pro- duction representative data for multiple high load applications. 2-1. 4 for statements with high arithmetic intensity are selected for each high load application. 2-2. 4 OpenCL with high arithmetic intensity loop are created and pre-compiled. Resource usage of each OpenCL are showed, then 3 OpenCL with high values of arithmetic intensity/resource usage are selected. 2-3. 3 OpenCL is measured with production representative data. Then, an OpenCL that combines 2 for statements with top 2 performances is created and additionally measured the performance as well. 2-4. The highest speed ofﬂoad pattern in 4 measurements is selected as a ﬁnal solution for each high load application. 3. The processing time of the current ofﬂoad pattern and the extracted multiple new ofﬂoad patterns are measured using production representative data, and the performance improvement effects based on the frequency of production use are calculated. 3-1. Calculation with the current ofﬂoad pattern (actual pro- cessing time reduction in veriﬁcation environment)*(frequency of production use). 3-2. Calculation with multiple new ofﬂoad patterns (ac- tual processing time reduction in veriﬁcation environ- ment)*(frequency of production use). 4. The reconﬁguration proposal is determined by the perfor- mance improvement effect of the new ofﬂoad pattern is more than the threshold of the current ofﬂoad pattern. 4-1. (3-2)/(3-1) of each high load application is calculated, and calculated values are checked if the value is higher than the threshold. If the value is more than the threshold, the reconﬁguration is proposed, and nothing is done if the value is below the threshold. 5. A reconﬁguration of FPGA is proposed to the contract user and the user responses OK or NG. 6. A static reconﬁguration is conducted by starting new OpenCL in a production environment. 6-1. New ofﬂoad pattern compilation. 6-2. Stop operation of the current ofﬂoad pattern. 6-3. Start the operation of the new ofﬂoad pattern. The method selects high load applications in step 1. For the current FPGA ofﬂoad application, the processing time is cal- culated by applying the improvement coefﬁcient to calculate where it is not ofﬂoaded to compare other applications which are processed CPU only. When choosing a representative data, the average data size may vary signiﬁcantly from the actual production data, I use the mode value of the data size. III. E VALUATION A. Evaluation conditions 1) Evaluated applications: The evaluated applications are mainly signal processing and image processing, which are expected to be used by many users in FPGAs. The time-domain ﬁnite-impulse response ﬁlter (tdFIR) for signal processing is a type of ﬁlter that cuts off the output whenan impulse function is input to the system in a ﬁnite time. There are various implementations, but the C code of [34] is used. When considering an application that transfers signal data from a device to a network in IoT or other situations, it is assumed that the data will be sent to the cloud after processing signals such as ﬁlters in order to reduce network costs. Therefore, I think that the automatic speed-up of signal processing in FPGA has a wide range of applications. MRI-Q [35] is an MRI image processing that calculates the Q-matric that represents the scanner settings for calibration. MRI-Q is used in a 3D MRI reconstruction algorithm in non- Cartesian space. In IoT or other situations, image processing is often required for automatic monitoring of camera images, and performance of image processing throughput is required to be enhanced. In the performance measurement at the time of ofﬂoad pattern extraction, MRI-Q performs 3D MRI image processing and depends on the data size, but in the assumed usage, the processing time is measured using 64*64*64 size data. In addition, the Himeno benchmark [36] for uncompressed ﬂuid analysis, Symm (Symmentry matrix manipulation) [37] for symmetric matrix calculation, and DFT (Discrete Fourier transform) [38] for discrete Fourier transform calculation are run on the same server and execution requests are received. 2) Evaluation methods: I conﬁrmed the proposed method. Before the start of operation, the user speciﬁes the ofﬂoad of tdFIR and automatically ofﬂoads it to FPGA. In the production environment, only tdFIR is ofﬂoaded to FPGA, and MRI- Q, Himeno Benchmark, Symm and DFT are run by CPU only processing. A request load is applied to the production environment server for a certain period of time, the requests are analyzed, a reconﬁguration to a new ofﬂoad pattern with a high performance improvement effect is proposed and the reconﬁguration is performed after user approval. The conditions for FPGA ofﬂoad are as follows. Ofﬂoad target: Number of loop statements. tdFIR 6, MRI-Q 16, Himeno 13, Symm 9, DFT 10. Narrow down of Arithmetic Intensity: Narrowing down to the top 4 loop statements in arithmetic intensity analysis of ROSE framework [39] Narrow down of resource efﬁciency: Narrow down to the top 3 loop statements in resource efﬁciency analysis Number of measured ofﬂoad patterns: 4 (The ﬁrst measure- ment measures 3 ofﬂoad patterns with high resource efﬁciency, and the second measurement measures the combination pattern of the 2 loop statement ofﬂoads that are high performance in the ﬁrst measurement.) The operational conditions for FPGA reconﬁguration are as follows. Request frequency: tdFIR 300 req/h, MRI-Q 10 req/h, Himeno 3 req/h, Symm 2 req/h, DFT 1 req/h requests are applied for 2 hours. Types of data: 3 types of data are prepared. Small size of sample data, large size of sample data, and double of large data which copies and adds large sample data once. In tdFIR and MRI-Q, requests for small, large and double large sizes Fig. 2. Evaluation environments are requested at a ratio of 3:5:2. Himeno, Symm and DFT are only requested the data which are same with equipped sample data. Long term during load analysis: 2 hours Short term when selecting representative data: 1 hour Number of high load applications: 2 Threshold of performance improvement effect: 2.0 During the reconﬁguration, the performance improvement effect and the processing time of each step associated with the reconﬁguration are acquired. 3) Evaluation environments: Intel FPGA PAC D5005 (Intel Stratix 10 GX FPGA, Logic Element 2,800,000) is used as the evaluation FPGA. The server equipped with Intel FPGA PAC D5005 is DELL EMC PowerEdge R740 (CPU: Intel Xeon Bronze 3206R * 2, RAM: 32GB RDIMM * 4). Intel Acceleration Stack Version 2.0 is used for FPGA control. By dividing the C language program into a kernel program and a host program according to the OpenCL syntax, FPGA ofﬂoad processing is performed by OpenCL, and reconﬁguration to another OpenCL program is also processed by Intel Acceler- ation Stack. Figure 2 shows the evaluation environment and speciﬁca- tions. Here, the note PC speciﬁes the application code to be ofﬂoaded, extracts the ofﬂoad pattern through performance measurement in the veriﬁcation environment, and then deploys it to the production environment. Execution requests are made to the production environment applications from the note PC periodically. The production environment requests are analyzed, new ofﬂoad patterns are extracted using the veriﬁca- tion environment, and after user conﬁrmation, the production environment is reconﬁgured into the new ofﬂoad pattern. B. Results Figure 3 shows the degree of improvement in the pro- cessing time of the ofﬂoad application before and after the reconﬁguration and the total processing time (corrected for the improvement coefﬁcient) for a certain period. First, tdFIR was ofﬂoaded before the reconﬁguration, the degree of improve- ment in the assumed data before the start of operation was 2.07, and the load of 300 req/h was applied after the start of Fig. 3. Comparison of performance improvement through reconﬁguration of the proposed method operation. 159 seconds which calculated from the total actual processing time of the request * 2.07 is the total corrected processing time, and 600 is the total number of uses. Next, MRI-Q has a load of 10 req/h after the start of operation. 549 seconds of the total actual processing time of the request is the total processing time, and 20 is the total number of uses. These two are the applications with the high load. These two searches for new ofﬂoad patterns using production representative data. The processing time for one time in the new ofﬂoad pattern is reduced from 0.266 seconds to 0.129 seconds for tdFIR and from 27.4 seconds to 2.23 seconds for MRI-Q. By multiplying the number of production uses, the processing time is reduced. The degree of performance improvement is 41.1 seconds/h for tdFIR and 252 seconds/h for MRI-Q. From Fig. 3, the change from tdFIR ofﬂoad to MRI-Q ofﬂoad increases the performance improvement by 6.1 times and exceeds 2.0, so reconﬁguration is proposed to the user. The size of the request analysis is small because only several hours data is analyzed in this time, but it will take longer in proportion to the size. This time, it takes about only 1 second for request analysis and production representative data selection, about 1 day for improvement effect calculation, and about 1 second for reconﬁguration. Regarding the ofﬂoad trial before the start of operation, the trial of the new ofﬂoad patterns searches during operation, and the time to compile the new ofﬂoad pattern one the production environment, since one FPGA compilation takes 6 hours or more, the number of measurements is 4 takes more than a day for one application. Most of the processing such as analysis, including the trial of the new ofﬂoad patterns searches, is performed in the background during application operation in the production environment, so there is no user impact. The only thing that can be conﬁrmed is that the break time of the application is necessary for production reconﬁguration. However, the static reconﬁguration of OpenCL takes about only 1 second, and there is almost no effect. If a shorter break time is required, it is possible to use the dynamic reconﬁguration function of FPGA vendors. I conﬁrmed the FPGA reconﬁguration according to the usage characteristics during operation by changing from tdFIR ofﬂoad to MRI-Q ofﬂoad during operation. Through the recon- ﬁguration, the degree of performance improvement increased above the threshold value, and it was shown that the break time was sufﬁciently short. IV. C ONCLUSION In this paper, as an element of it, I have proposed an FPGA reconﬁguration method that reconﬁgures the appro- priate FPGA logic during operation according to the usage characteristics after the application operation starts. Before starting operation, the application loop statement is automatically ofﬂoaded to the FPGA. In the proposed method, the applications with large CPU processing times are analyzed from the actual request data at regular intervals, and the corresponding representative test cases are gathered. Next, the ofﬂoad patterns that speed up representative test cases are extracted through trial measurement in the veriﬁcation environment for large load applications. This is almost the same as ofﬂoad before the start of operation. Next, the processing time of the current ofﬂoad pattern and the extracted new ofﬂoad pattern are measured, and the processing time improvement based on the frequency of production use is calculated. Here, if the new ofﬂoad pattern has an effect greater than the threshold of the current ofﬂoad pattern, our method proposes to the users to carry out reconﬁguration. Once user consent is obtained, our method reconﬁgures FPGA logic us- ing OpenCL reconﬁguration in a production environment. The application that was automatically ofﬂoaded to the FPGA, and the FPGA logic was reconﬁgured during operation to another application in the experiment. The reduction in processing time was improved by reconﬁguration, and reconﬁguration was performed with a short interruption time of about 1 second. Thus, the effectiveness of the proposed method was conﬁrmed.