## 2 Materials and Methods
### Image Processing Pipeline  
Several additional external inputs were required for our forecasting algorithms: distortion parameters for the fish eye lens, zenith angle, True North, and most importantly the cloud base height (CBH). These inputs are used in the image processing pipeline (Figure 13) to output real-time GHI forecasts for the MGMS. A summary of the pipeline is included here, for details see [8]. (1) Distortion Removal (due to fish eye lens), (2) Cropping and Masking, (3) Calculation of \"Red-to-Blue Ratio\" (RBR), (4) Apply Median Filter to remove impulsive noise, (5) Thresholding to determine cloud presence, (6) Compute Cloud Cover percentage (clear/moderately-cloudy/overcast), (7) Project Clouds to height of CBH, (8) Use Optical Flow to move clouds forward in time, (9) Ray-Tracing to locate cloud shadows, and finally (10) Calculate GHI using shadow locations. Although physically correct, Step (9) Ray-Tracing is an inverse problem mathematically, hence \"ill-posed\". Small errors in locating shadows can produce significant errors in the forecast irradiance. To address this issue, we investigated using artificial intelligence and neural works to predict GHI values directly from forecast cloud locations.  
At NREL another raw image was acquired every 15-seconds. The pipeline described above must be fine-tuned if the processing SBC is to achieve the necessary throughput. An SBC is much more limited than a desktop server as regards CPU speed and available memory/storage, which is provided by a 32 Gb micro-SD card. The usual tradeoffs between keeping a large array in memory versus writing it to disk, are still present even though there is no disk. Efficient programming constructs are required if the goal of low cost is to be achieved. The EMS may run on a military grade RuggedCom server but the SkyImager software is constrained run on an ARM architecture. Steps in the pipeline that have little effect on the overall forecast accuracy can be eliminated. Profiling/timing runs on the optical flow algorithms will show bottlenecks that can be addressed. This is important whether a ray-tracing approach or a machine learning strategy is employed.  
The goal in intra-hour solar forecasting is real time PV power predictions. Those forecasts result from a two-step process: predicting cumulus cloud locations 15-min in the future and using projected cloud locations to forecast irradiance. Each step introduces errors. Work is ongoing for _Step 1 - Optical Flow_: compute error metrics of the 15-min ahead image versus the actual image. _Step 2 - Machine Learning_ takes the predicted image and computes GHI. This approach separates optical flow [50] from machine learning (ML) and allows _GHI to be predicted directly from the image itself_. The SkylImager can be used as a pyranometer for measuring/observing irradiance.  
Figure 12: (a) Hardkernel’s WeatherBoard, (b) PiNoIR camera, and (c) MQ 131 Ozone Sensor.  
The training datasets for the neural networks are region-specific, if not site-specific, and require many all-sky images taken on moderately cloudy days. The weights determined for Golden, CO, will have to be fined tuned for deployment in the San Antonio area for example. Training is computationally expensive, whereas the inference or prediction is very fast and can be handled by the Pi. Research on massive deep learning networks requires Graphical Processing Units (GPU) for training and software such as Theano, Keras, or Tensorflow.