## 6 Evaluation
### Evaluation Setup  
To evaluate our algorithm, we need a high-quality web service application. Gathering weather data from a collection of weather stations, matching the location of real-world telescopes has several issues. First, telescopes tend to be located at a few high-altitude sites with exceptional visibility; this makes for a small number of weather stations. Second, there are large gaps where the automated weather observation station (AWOS) does not record the weather conditions, because they are unserviceable. This results in a dataset that is too small to be used for validating our algorithms effectively. A better strategy is to use data for stocks tracked, for example, by the S&P 500 index. In particular, we used stock tracking data for the period of 1 January 2009 to 22 November 2011, almost two years' worth of data. A stock can be considered as \"services to your money.\" The data collected includes the low, high and closing price for the stock, as well as transaction volumes for each trading day; we use these as proxies for dynamic non-functional telescope attributes. We also tracked the Dow Jones Industrial Average (DJI) over the same period, which we use as an additional attribute when evaluating service quality (stock desirability). The full dataset is available on GitHub at [[https://github.com/Atousa/DYNAMIS](https://github.com/Atousa/DYNAMIS)]([https://github.com/Atousa/DYNAMIS](https://github.com/Atousa/DYNAMIS)). Our evaluation targets the use-case of a consumer searching for a web service that provides functionality needed in some application being developed. The individual services that could be chosen are in fact proxied by the various stocks that we tracked. This allows us to compare some 500 services (stocks) by evaluating several attributes varying with high frequency, such as price (P) and trading volume (V), and to compare service selection strategies (stock selection). Note that the analysis should not be considered investment advice. After gathering data, we compute the following additional attributes to be used during evaluation: change in price (\(\Delta\)P), change in volume (\(\Delta\)V) and relative strength _vs_. Dow Jones index (RSD); \(d\) and \(d-1\) are a specific trading date and its immediate predecessor respectively, while \(s\) represents a specific stock in our dataset. Formally, we define:  
\[\Delta\mathrm{P_{s}} =\frac{\mathrm{P_{d}-P_{d-1}}}{\mathrm{P_{d-1}}}\] \[\Delta\mathrm{V_{s}} =\frac{\mathrm{V_{d}-V_{d-1}}}{\mathrm{V_{d-1}}}\] \[\mathrm{RSD_{s}} =\Delta\mathrm{P_{s}}-\Delta\mathrm{P_{DII}}\]  
We now demonstrate how our Dynamis approach increases the effectiveness of selection by providing finer filtering and ranking of the services in the registries. The evaluation task is to rank a list of web services according to their relevance for a given consumer request (identify stocks that could generate profits). Using the data collected, we evaluate the quality of our picks by looking at subsequent profits or losses of the stocks selected, thus avoiding human judgement conflicts and their consequences in the evaluation result (_cf._, Section 6) [37]. Most of the analysis is illustrated using one stock from the technology sector: INTC (Intel Corp.); but we also look at AAPL (Apple Inc.) and GOOG (Google Inc.) when we need to compare stock performance. For clarity, the data for a given stock is shown consistently in the same color throughout this section. Detailed results for all of the stocks tracked are available at GitHub [[https://github.com/Atousa/DYNAMIS/](https://github.com/Atousa/DYNAMIS/)]([https://github.com/[PERSON]/DYNAMIS/](https://github.com/Atousa/DYNAMIS/)). Figure 1 shows significant variability in the trading volume data from one day to the next; reported here is the percentage change with respect to the previous trading day's transaction volume (\(\Delta\)V). However, as shown in Figure 2, by computing the histogram of the data over a period of time, some of the information contained therein becomes more readily visible.