Reducing electricity cost through virtual machine placement in high performance computing clouds In this paper, we first study the impact of load placement policies on cooling and maximum data center temperatures in cloud service providers that operate multiple geographically distributed data centers. Based on this study, we then propose dynamic load distribution policies that consider all electricity-related costs as well as transient cooling effects. Our evaluation studies the ability of different cooling strategies to handle load spikes, compares the behaviors of our dynamic cost-aware policies to cost-unaware and static policies, and explores the effects of many parameter settings. Among other interesting results, we demonstrate that (1) our policies can provide large cost savings, (2) load migration enables savings in many scenarios, and (3) all electricity-related costs must be considered at the same time for higher and consistent cost savings. Cloud computing is rapidly growing in importance as increasing numbers of enterprises and individuals are shifting their workloads to cloud service providers. Services offered by cloud providers such as Amazon, Microsoft, IBM, and Google are implemented on thousands of servers spread across multiple geographically distributed data centers. There are at least three reasons behind this geographical distribution: the need for high availability and disaster tolerance, the sheer size of the computational infrastructure, and the desire to provide uniform access times to the infrastructure from widely distributed user sites. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. SC 11, November 12-18, 2011, Seattle, Washington, USA Copyright 2011 ACM 978-1-4503-0771-0/11/11 ...$10.00.. The electricity costs involved in operating a large cloud infrastructure of multiple data centers can be enormous. In fact, cloud service providers often must pay for the peak power they draw, as well as the energy they consume. For example, some data centers are designed to consume a maximum of 50MW of power, which is equivalent to a city with 40K households. Depending on utilization, the energy costs alone (not including the peak power costs) of a single such data center can easily exceed $15M per year. This cost can almost double when peak power is considered.. Lowering these high operating costs is one of the challenges facing cloud service providers. Fortunately, the geographical distribution of the data centers exposes many opportunities for cost savings. First, the data centers are often exposed to different electricity markets, meaning that they pay different energy and peak power prices. Second, the data centers may be placed in different time zones, meaning that one data center may be consuming off-peak (cheap) electricity while another is consuming on-peak (expensive) electricity. Finally, the data centers may be located in areas with widely different outside temperatures, which have an impact on the amount of cooling energy used.. Given the different characteristics of the data centers' energy consumptions, energy prices, and peak power prices, it becomes clear that we can lower operating costs by intelligently placing (distributing) the computational load across the wide area. This is exactly the topic of this paper. In particular, we consider services such as Amazon's EC2, which implements "infrastructure as a service" (IaaS). In IaaS services, users submit virtual machines (including the user's entire software stack) to be executed on the physical machines of the provider. Users are responsible for the management of their virtual machines, whereas the provider can potentially decide where the virtual machines should execute. IaaS services can host different user applications, ranging from interactive services to large high-performance computations (HPC). In this paper, we focus on IaaS services that support HPC workloads (e.g., [2, 12] ).. In this context, our study proposes policies for virtual machine placement and migration across data centers. The policies are executed by front-end devices, which select the destination for each job (possibly a collection of virtual machines) when it is first submitted. Later, a (possibly different) front-end may decide to migrate the (entire) job to another data center. We assume that users provide an estimate of the running time of each job. Using this estimate, the front-end can estimate the cost of executing the job at each data center. Specifically, our policies model the energy costs, peak power costs, the impact of outside temperature on cooling energy consumption, and the overhead of migration. For comparison, we also consider policies that are cost-unaware, such as Round Robin, and a policy that is cost-aware but static.. An important aspect of our load placement approach is that a change in electricity price at a data center (e.g., when the data center transitions from on-peak to off-peak prices) could cause a significant increase in its load in just a short time. As cloud service providers operate their data centers closer to the overheating point (when the servers' inlet air reaches an unsafe temperature) to reduce cost, such load increases can compromise the ability of the data centers to adjust their cooling before servers start to turn themselves off. This problem is exacerbated if providers turn off the water chillers (when not needed) to further reduce cost. Chillers typically incur significant delays to become fully effective for cooling when they are turned on. We study this effect and its implications for data center cooling and our policies. In particular, our policies predict changes in the amount of load offered to the data centers, and start "pre-cooling" them if necessary to prevent overheating.. Our evaluation uses detailed simulations. Our study of cooling effects relies on Computation Fluid Dynamics simulations of temperatures in realistic data centers. Increasingly, cloud service providers are running their data centers at higher temperatures (i.e., closer to the overheating point) to reduce cooling energy consumption [28] . Our study shows that under such scenarios, large and rapid changes in load can indeed produce overheating. The study also shows that certain cooling strategies are substantially worse than others, and require longer periods of pre-cooling to prevent overheating. Finally, the study illustrates the effect of outside temperature on the energy consumption of the cooling infrastructure.. Our study of the load placement policies relies on event-driven simulations of a network of data centers using real workload traces. The results of this study show that our policies achieve substantially lower costs than their competitors. Migration provides non-trivial cost benefits, as long as the amount of data to migrate is not excessive (in which case the policies decide not to migrate). Moreover, our results show that it is critical for our policies to consider their three cost components: server energy, peak power, and cooling energy. Finally, our results explore the impact of different values for our key simulation parameters, including different electricity prices and outside temperatures.. We conclude that cost-aware load placement policies can lower operational costs significantly for cloud service providers. However, these policies must properly account for the significant changes in load that they may cause and how those changes may affect the provider's cooling infrastructure. Our policies are a strong first step in these directions.. In summary, our main contributions are:. • A demonstration of the transient effect of large and rapid increases in load on the cooling of a data center; • A study of the behavior of many cooling strategies and the effect of different levels of pre-cooling on them; • Effective policies for load placement in HPC cloud services that employ multiple data centers. The policies consider all power-related costs, as well as transient cooling effects; and • Extensive results on the behavioral, energy, and cost implications of those policies. Dynamic cost-aware load distribution can lead to significant cost savings. The actual savings depend on many parameters that determine how much dynamic cost diversity exists, and the flexibility with which policies can affect load distribution (e.g., if there is little spare capacity then there is almost no flexibility for load distribution). Intuitively, awareness of cooling is most important when the energy price and/or the peak power charge are only slightly lower at a hot location than at a cool location. In this case, not considering the cooling cost can lead to large cost and energy usage penalties when load distribution policies greedily choose the cheapest prices. Similarly, awareness of peak power charges is most important when the ratios of peak power charges to energy prices are widely different across time and/or data centers. However, to correctly account for all possible different scenarios, it is critical that the dynamic load distribution policy considers all three cost components. In this paper, we have studied the possibility of lowering electricity costs for HPC cloud providers that operate multiple geographically distributed data centers. Specifically, we designed policies that intelligently place and migrate load across the data centers to take advantage of time-based differences in electricity prices and temperatures. Our policies account for three important electricityrelated costs: energy price, peak power charge, and the energy consumed by the cooling system. We developed a model of data center cooling for a realistic data center and cooling system. We simulated the model to obtain the cooling power as a function of data center load and outside temperature. This allowed us to investigate the impact of cooling on total cost, and explore whether cooling-aware load distribution can lead to cost savings. We also studied transient cooling effects resulting from abrupt, large changes in data center loads. We concluded that pre-cooling is necessary to prevent overheating in these scenarios. Our policies incorporate this pre-cooling.. Finally, we have shown that intelligent placement and migration of load can indeed lead to significant cost savings. Further, all electricity-related costs must be considered to maximize and ensure consistent cost savings. We use simulation to evaluate our framework and policies. Our multi-data center simulator takes as input an HPC workload trace, energy price traces, peak power charges, and temperature traces. Using these inputs, it simulates a load distribution policy and the resulting job completion times and data center energy consumptions. Our simulator accounts for the base energy of servers that are on (and turning-on/off), dynamic energy consumption, cooling levels needed to support specific data center loads vs. outside temperatures (cooling energy), delays in turning chillers on, and the resource usage (cores, memory, power demand, and migration footprints) of every job at a one-second granularity. Our evaluations are based on week-long traces, as well as sensitivity studies varying our main simulation parameters.. Workload. We use a trace of jobs submitted to an HPC cluster available from the Parallel Workloads Archive [14] . This trace contains approximately four months (December 1999 through April 2000) of data derived from the accounting records of the LSF software running on a 2048-node Origin 2000 cluster (Nirvana) at Los Alamos National Lab. The trace, LANL-O2K, contains a record for each job serviced by the cluster, with each record containing a job number, submitted time, actual run-time, and maximum number of cores and amount of memory used.. We extract a random week from this trace (January 24-31, 2000) and use it as the workload for all experiments discussed below. This week-long trace contains 6680 jobs, with a peak processing demand of 3867 cores. The maximum memory requirement of any one job is 4GB. Figure 5 shows the number of CPUs demanded by active jobs and Figure 6 shows the CDF of job run-times for our workload trace. There is a significant number of short running jobs (almost 30% run for less than 10 seconds) but there are also many long running jobs (>20% run for 1 hour or longer).. To map the above workload to our environment, we assume that each job can be split into VMs, one single-core VM per each core used by the job. The memory used by a job is equally divided among its VMs. When migrating a job, we assume that only the memory images of the VMs and a small amount of persistent data, expressed as the diffs of the data set [52] , need to be transferred to the target data center.. As previously discussed, each arriving job specifies the number of VMs needed and an estimated run time. The service SLA is to complete each job within the estimated run-time plus 5% of the job's total estimated processing time (estimated run-time×number of processors) plus 30 seconds. The latter part of the slack is to avoid missing the SLA for short running jobs when machines need to be turned on to accommodate them.. Data centers. We simulate a single front-end located on the East Coast of the US in most cases. The front-end distributes requests to three equal-size data centers located in the US West Coast (Northern California), US East Coast (Georgia), and Europe (Switzerland), respectively. As we focus on the cost savings enabled by geographical distribution, we assume that the data centers have the same PUE except for the cooling energy required by their outside temperatures. (Our framework can be easily extended to account for data centers with different PUEs and cooling technologies.) We Table 3 : Electricity prices [1, 13, 15, 20, 32, 41, 42] .. assume that the users' persistent data are replicated across the three data centers, so that each job can be placed at any data center that has sufficient idle capacity to host the entire job. Similar to [52] , we assume an inter-data-center bandwidth of 464 Mbps.. Provisioning each data center to be consistent with our modeled data center in Section 3 gives us a total of 5760 cores (1440 servers) across the three data centers. This corresponds to approximately 49% spare capacity at the peak load. This provisioning is consistent with current practices and allows the service to support the workload even if one of the data center fails.. Within a data center, each server has 4 cores and 4GB of memory. In our workload trace, no single VM requires more than 1GB of memory. Thus, job placement is essentially a core selection problem, i.e., a job can be placed at any data center that has enough idle cores. Within a data center, each VM of a job can be placed on any idle core. In our experiments, jobs are placed within a data center to minimize the number of active servers.. Each server's base power demand is 200W, with a peak power demand of 300W (corresponding to 100% utilization). Turning a machine on and off both require 30 seconds. Each data center turns off machines, if its has over 20% idle capacity. If not needed to maintain spare capacity, machines are turned off after they have been idle for 60 seconds.. Cooling. We use the cooling model developed in Section 3 to compute the energy consumed by the cooling system at each data center. Each data center adjusts its cooling system so that temperatures inside the data center never exceed 30 • C. Jobs are never run before the cooling system can accommodate the increased heat. Specifically, if a chiller has to be turned on, arriving jobs are delayed until the chiller is ready (20 minutes). The front-end can ask data centers to pre-cool (by turning on the chiller early) in preparation to receive load; this is used in our cost-aware with migration policy. The chiller is turned off after it has not been used for 40 minutes.. Electricity prices. We simulate two sets of electricity prices at each data center, one for "on-peak" hours (weekdays from 8am to 8pm) and another for "off-peak" hours (weekdays from 8pm to 8am and weekends) [1, 13, 41] . The on-peak prices, listed in Table 3 , are obtained either from previous publications [23, 32] or from information listed on power utility providers' Web sites [15, 20, 42] ; by default, the off-peak prices are 1/3 of the on-peak prices.. Outside temperatures. We collected historical weather data from the Weather Underground Website [50] for the week of May 1-7, 2010. This period is interesting because not all locations are hot enough to require the chiller to be on all the time, and not all of them are cold enough to depend on just free cooling. Figure 7 shows the outside temperatures for our simulated period. Note that the East Coast location is quite a bit hotter than the other two locations but has slightly lower energy prices.

O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O B O O O O O O O O B O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O B O B O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O B O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O


Choosing software metrics for defect prediction: an investigation on feature selection techniques The selection of software metrics for building software quality prediction models is a search-based software engineering problem. An exhaustive search for such metrics is usually not feasible due to limited project resources, especially if the number of available metrics is large. Defect prediction models are necessary in aiding project managers for better utilizing valuable project resources for software quality improvement. The efficacy and usefulness of a fault-proneness prediction model is only as good as the quality of the software measurement data. This study focuses on the problem of attribute selection in the context of software quality estimation. A comparative investigation is presented for evaluating our proposed hybrid attribute selection approach, in which feature ranking is first used to reduce the search space, followed by a feature subset selection. A total of seven different feature ranking techniques are evaluated, while four different feature subset selection approaches are considered. The models are trained using five commonly used classification algorithms. The case study is based on software metrics and defect data collected from multiple releases of a large real-world software system. The results demonstrate that while some feature ranking techniques performed similarly, the automatic hybrid search algorithm performed the best among the feature subset selection methods. Moreover, performances of the defect prediction models either improved or remained unchanged when over 85% of the software metrics were eliminated. Defect prediction is an important process activity in software engineering practice. A verified strategy used for this task is building software quality prediction models that estimate the quality of program modules, e.g. fault-prone (fp) or not-fault-prone (nfp) [1] [2] [3] [4] [5] . A practitioner can apply such a model toward implementing a targeted and prioritized software quality improvement activity. This approach allows for a more cost-effective usage of the limited project resources.. Software attributes can characterize the software quality of both the product and the process of software development [6] . Attributes of software quality, such as defect density and failure rate, are external measures of the software product and its development process. We focus on utilizing software metrics, such as code-level measurements and defect data, to build defect predictors or software quality models. This is based on the practical assumption that these software metrics will 580 K. GAO ET AL. capture the quality of the end product. By exploring knowledge stored in historical project data, characterized in the form of software metrics and defect data, one can build effective software quality prediction models.. Generally, a software quality prediction model is built using software metrics and defect data collected from a previously developed system release or similar software project. Upon validation of such a model, it is ready for predicting the fault-proneness of program modules that are currently under development. A low quality or fp prediction can justify the application of available quality improvement resources to those programs. In contrast, an nfp prediction can justify non-application of the limited resources to these already high-quality programs. The goal is to achieve high software reliability and quality with an effective use of the available resources.. Since a software quality prediction model is based on the knowledge stored in the known software metrics, the selection of the specific set of metrics becomes an integral component of the model-building process [6] . In other words, the quality of the software measurement data plays an important role in defect prediction. While not the focus of this study, reducing the different types of noise in software measurement data has been addressed in the literature [1, 4, 7] . The primary focus of this paper is on feature selection (or attribute selection) of software metrics for defect prediction, and thereby, improving the quality of the software measurement data.. We approach the problem from a search-based perspective, i.e. explore the available software metrics' space and derive a useful set of software metrics. This search is a preprocessing step to building the software defect prediction models. Feature selection algorithms can be categorized into two groups, feature ranking and feature subset selection [8] . Feature ranking assesses each individual feature (software metrics) according to some criterion and then the analyst selects some of the features that are appropriate for a given data set. In contrast, feature subset selection searches for subsets of attributes that collectively have good predictive capability. This is based on the assumption that a given attribute may have better predictability when combined with some other attributes, as compared to when used by itself. However, one of the problems for subset selection is that the number of candidate subsets may be prohibitively large and evaluating all of these is impractical. In this paper, we propose a hybrid attribute selection approach consisting of feature ranking followed by feature subset selection. Feature ranking prioritizes features and selects a pre-specified number of relevant features to reduce the search space. Subsequently, a search algorithm is applied to the reduced search space and feature subset selection is used to choose the final attribute subset.. In the context of software quality prediction, we empirically investigate seven different feature ranking techniques: chi-square (CS), information gain (IG), gain ratio (GR), Kolmogorov-Smirnov statistic (KS), two forms of the Relief algorithm (RLF), and symmetrical uncertainty (SU). The results of feature ranking are then analyzed by three different feature subset selection methods (algorithms): exhaustive search (ES), heuristic search (HS), and automatic hybrid search (AHS). We also consider no subset selection algorithm (referred to as NONE), thus, providing four techniques for attribute subset searching. We note that the KS feature ranking technique and the AHS feature subset selection algorithm were proposed by our team in earlier works [9, 10] . Our approach combines both, feature ranking and feature subset selection, for hybrid search-based approach for selecting the appropriate software metrics. Subsequent to attribute selection, we consider five different learners for training defect predictors [11] : naïve Bayes (NB), multilayer perceptron (MLP), support vector machine (SVM), logistic regression (LR), and k nearest neighbor (KNN).. A comparative analysis based on statistical computations suggests that software quality prediction models based on the various feature ranking techniques have similar performances, except for the CS method. In addition, the AHS feature subset selection algorithm yielded a better predictive performance than the other search algorithms. A further empirical analysis demonstrates that even after removing over 85% of the software metrics from the original data, model performances remained similar or even improving in some cases. It is shown that a reduced search space (of software metrics) does not deteriorate the outcomes of the feature subset selection methods, nor does it adversely affect the predictive performances of the classification models. In fact, the subsets of software attributes obtained from the reduced search spaces are more relevant/associated to the class (fault-proneness label) attribute. The selection of software metrics for building software quality prediction models is a search-based software engineering problem. An exhaustive search for such metrics is usually not feasible due to limited project resources, especially if the number of available metrics is large. Defect prediction models are necessary in aiding project managers for better utilizing valuable project resources for software quality improvement. The efficacy and usefulness of a fault-proneness prediction model is only as good as the quality of the software measurement data. This study focuses on the problem of attribute selection in the context of software quality estimation. A comparative investigation is presented for evaluating our proposed hybrid attribute selection approach, in which feature ranking is first used to reduce the search space, followed by a feature subset selection. A total of seven different feature ranking techniques are evaluated, while four different feature subset selection approaches are considered. The models are trained using five commonly used classification algorithms. The case study is based on software metrics and defect data collected from multiple releases of a large real-world software system. The results demonstrate that while some feature ranking techniques performed similarly, the automatic hybrid search algorithm performed the best among the feature subset selection methods. Moreover, performances of the defect prediction models either improved or remained unchanged when over 85% of the software metrics were eliminated. Copyright q 2011 John Wiley & Sons, Ltd. Software metrics have a very important role in software development and software quality assurance activities. A software quality prediction model is often based on knowledge extracted from software measurement data. The quality of the underlying software measurement data is thus critical. An intelligent selection of software metrics before training a defect prediction model is likely to improve the end result-by removing redundant and less important features. We present an extensive investigation of feature selection for software quality modeling.. We studied seven filter-based feature ranking techniques and three filter-based subset selection search algorithms. Thus, each ranking technique is associated with each search algorithm. The search space for the subset selection algorithms is reduced for a more practical application of the proposed approach. A direct search approach is also evaluated, where the top six software metrics from a given ranking are selected as the feature subset. Software quality classification models are built subsequent to each feature selection process. We also look into the impact of a reduced search space on the feature subset selection outcome. A feature subset is obtained with the AHS search algorithm on the original data (42 metrics), and then compared with the corresponding reduced search space outcomes.. The empirical results, based on a large real-world software system, demonstrate that among the different search algorithms, our AHS approach generally provides better performances. Among the seven feature ranking techniques, CS has consistently poor performances. It was also observed that reducing the attribute search space (for practical reasons) did not have an adverse effect on either subset selection or on classification results. This is important since a smaller search space lends to a more quicker overall software quality modeling process. Among the five learners, NB, MLP, and LR performed better than SVM and KNN. With the exception of SVM, all the learners we considered are very commonly used in the software engineering community. A final inference is that even after removing 85% of the available number of software metrics the defect prediction models were not adversely affected; in fact, in some cases the results were better. This is another important point for software practitioners, since practitioners prefer fewer software metrics from the data collection, management, modeling, and analysis points of view.. The future work will involve conducting additional empirical studies with data from other software projects and application domains. Object-oriented systems and object-oriented metrics will be considered in our future research. An investigation of wrapper-based feature selection will provide a new perspective on filter-based approaches for the software engineering community. In addition, modeling the cost or effort of the different feature selection techniques is an important topic that will be studied in the future. There has been an increase in work on Search Based Software Engineering (SBSE) techniques for solving optimization problems, including requirements, project planning, maintenance, and re-engineering. Harman et al. [12, 13] provide a comprehensive survey on SBSE-related studies. In this paper, we provide a unique insight into feature selection for obtaining a good set of software metrics for defect prediction. To our knowledge, very little work has been done on feature selection for software quality prediction modeling.. Rodríguez et al. [4, 14] applied feature selection with three filter models and three wrapper models to five software engineering data sets. They conclude that the reduced data sets maintained the prediction capability with fewer attributes than the original data sets. In addition, while it was stated that the wrapper model was better than the filter model, it came at a high computational cost. In contrast to our study, their conclusions were based on evaluating models using cross validation instead of an independent test data set. It is known in the software engineering and data mining communities that prediction models are best evaluated based on their generalization performance, i.e. using a test data set. This is important for avoiding an overfitting of the prediction model to the training data set. In our case studies, three independent validation (test) data sets are used-as discussed in the later sections. Chen et al. [15] have studied feature selection using wrappers in the context of software cost/effort estimation, and conclude that the reduced data set could improve the estimation.. Feature selection is an important data preprocessing task in many data mining and machine learning applications. Liu and Yu [16] provide a survey of feature selection algorithms and present an integrated approach to intelligent feature selection. Guyon and Elisseeff [8] outline the key approaches used for attribute selection, including feature construction, feature ranking, multivariate feature selection, efficient search methods, and feature validity assessment methods. Hall and Holmes [7] investigated six attribute selection techniques that produce ranked lists of attributes and applied them to 15 data sets from the UCI repository. They conclude that while the ranking techniques yielded similar results, a wrapper/filter approach to attribute selection was the best.. Many applications of feature selection in various fields have been reported [17] [18] [19] . Jong et al. [20] introduced methods for feature selection based on SVMs. Ilczuk et al. [21] highlighted the importance of attribute selection in judging the qualification of patients for cardiac pacemaker implantation. In the context of text mining, where attributes are binary in value, Forman [18] investigates multiple filter-based feature ranking techniques. Feature ranking techniques assess attributes individually based on a given criterion and order them accordingly. However, it was found that sometimes an attribute may be less useful by itself but can make a significant contribution when combined with other attributes. Feature subset selection approaches consider this issue by searching and selecting subsets of attributes that collectively have good performance. Attribute selection can also be divided into wrappers and filters [16] . Wrappers are algorithms that use feedback from a learning algorithm to determine which attributes to use in building a predictive model. In filters, the training data are analyzed using a method that does not require a learning algorithm to determine which attribute(s) are most relevant. In this paper, we only consider filters instead of wrappers for feature selection. The reasons include that 585 (1) the use of wrappers would be too complex for inclusion in our case study; (2) the complexity of the problem was also one of the reasons why we intelligently reduced the attribute space during feature selection; and (3) the dependency of wrappers on a specific learner makes it difficult to find the best suitable wrapper since there are many learners available to choose from.

O O O O B I O O O O O O O O O O O O O B I O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O B I O O O O O O O O O O O B I O O O B I O O O O O O O O O O O B I I O O O O O B I O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B I I O O O O O B I O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O B I O B I I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O B I O O B I I B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O B I O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O B I I O O O O O O O O O O O O O O O O B I O B I I O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B I O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O B I O O O O O O O O O O O B I O O O B I O O O O O O O O O O O B I I O O O O O B I O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B I I O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O B I O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Site level noise removal for search engines The currently booming search engine industry has determined many online organizations to attempt to artificially increase their ranking in order to attract more visitors to their web sites. At the same time, the growth of the web has also inherently generated several navigational hyperlink structures that have a negative impact on the importance measures employed by current search engines. In this paper we propose and evaluate algorithms for identifying all these noisy links on the web graph, may them be spam or simple relationships between real world entities represented by sites, replication of content, etc. Unlike prior work, we target a different type of noisy link structures, residing at the site level, instead of the page level. We thus investigate and annihilate site level mutual reinforcement relationships, abnormal support coming from one site towards another, as well as complex link alliances between web sites. Our experiments with the link database of the TodoBR search engine show a very strong increase in the quality of the output rankings after having applied our techniques. The popularity of search engines has thoroughly increased over the past years. And so has the amount of information they are indexing. At the same time, upon searching this overwhelming quantity of data, people usually view only the top answers returned for each query [20] . It is thus very important to provide these responses with the best quality possible. Alas, currently this is not an easy task.. Copyright is held by the International World Wide Web Conference Committee (IW3C2). Distribution of these papers is limited to classroom use, and personal use by others. Search engines adopt several different sources of evidence to rank web pages matching a user query, such as textual content, title of web pages, anchor text information, or the link structure of the web. This latter measure is one of the most useful sources of evidence adopted. To extract information from the link structure, search engines use algorithms that assess the quality (or popularity) of web pages by analyzing the linkage relationships among them. The success of this strategy relies on the assumption that a link to a page represents a vote from a user that sustains the quality of that page.. In spite of the success of link analysis algorithms, many link structures created using web hyperlinks lead these algorithms to provide wrong conclusions about the quality of web pages. This phenomenon happens because links that cannot be interpreted as votes for quality sometimes negatively affect the link analysis results. We name these here as noisy links. A subset of them has already been addressed exhaustively in prior work, namely the nepotistic links (also called spam links), i.e., the links intentionally created to artificially boost the rank of some given set of pages, usually referred to as spam pages [27] . In fact, given the importance of search engines in modern society, many online organizations currently attempt to artificially increase their rank, since a higher rank implies more users visiting their pages, which subsequently implies an increased profit. Thus, a new industry specialized in creating spam information has emerged, called Search Engine Optimization (SEO) [17] .. Even though it is very important to diminish the noise effect induced by spammers (i.e., creators of spam), many other noisy links may appear in the web graph and should be detected as well. For instance, two sites from companies of the same group may be strongly interconnected by links that do not represent votes for quality. Such links are created due to a relationship between the entities represented by both sites and not by the intention to vote for the quality of their pages, as assumed by the link analysis algorithms. In fact, we argue that most of the noisy links on the web are created in a non-intentional way. Therefore, devising algorithms to detect noise in general (which also includes spam) is better than devising algorithms considering only spam attacks over the web graph.. In this paper we propose a site-level approach for detecting generic noisy links on the web. Previous algorithms have focused on identifying noise only by analyzing page level relationships, which clearly misses some of the higher level noise, generated between a group of sites. We investigate three main types of site level relationships: mutual reinforcement (in which many links are exchanged between the two sites), abnormal support (where most of one site's links are pointing to the same target site), and link alliances (in which several sites create complex link structures that boost the PageRank score of their pages). When the relation between such sets of sites is considered suspicious, we assume that the links between them are noisy and penalize them accordingly. Finally, it is important to note that this new approach is complementary to the existing page level approaches, and both strategies should be adopted simultaneously for identifying noisy links in a search engine database.. We studied the impact of our noise removal techniques on PageRank [25] , as it is probably the most popular link analysis algorithm for computing the reputation of pages over the web. Our experiments have shown an improvement of 26.98% in Mean Reciprocal Rank [19] for popular bookmark queries, 20.92% for randomly selected bookmark queries, and up to 59.16% in Mean Average Precision [4] for topic queries. Furthermore, 16.7% of the links from our collection were considered as noisy, while many of them could not be considered nepotistic, thus demonstrating the importance of searching for noisy links in search engine databases instead of searching only for spam.. The paper is organized as follows: Section 2 presents some background information and discusses the related work. Section 3 describes our site-level noise detection approach and the algorithms we proposed. We extensively evaluate these algorithms and analyze their performance in Section 4. Finally, Section 5 draws some conclusions and discusses future work. In this paper we proposed using site level link analysis to detect the noisy links from search engine link databases. We designed and evaluated algorithms tackling three types of inappropriate site level relationships: mutual reinforcement, abnormal support and link alliances. Our experiments have showed an improvement of 26.98% in Mean Reciprocal Rank for popular bookmark queries, 20.92% for randomly selected bookmark queries, and up to 59.16% in Mean Average Precision for topic queries. Furthermore, our algorithms identified up to 16.7% of the links from our collection as noisy, while many of them could not be considered nepotistic, thus demonstrating that searching for noisy links in search engine databases is more important than searching only for spam.. While most of the algorithms we presented in this paper directly removed the identified malicious links, in future work we intend to investigate using different weights for various types of links, according to the relation they represent (i.e., inter-site or intrasite relation), as well as to their probability of representing a vote of importance. Additionally, we would like to study more complex (eventually automatic) approaches to tune up the parameter thresholds, instead of using the MRR scores resulted for bookmark queries. Bookmark Queries. We evaluated the bookmark queries in terms of Mean Reciprocal Rank (MRR) and Mean Position (MPOS) of the first relevant URL output by the search engine. Table 2 shows the MRR scores for each algorithm with popular bookmark queries. The best result was achieved when combining all Method Threshold UMSR 250 BMSR 2 SLAbS 2% 3 for each database, when compared to the ALL LINKS version (i.e., PageRank on the original link database). The only method that had a negative impact on MRR was the UMSR, which indicates that many unidirectional relations between sites are rather useful for the ranking. This was also the only algorithm for which the T-test indicated a non-significant difference in the results (p-values lower than 0.25 are taken as marginally significant, lower than 0.05 are taken as significant, and lower than 0.01 as highly significant). Table 3 presents the Mean Position of the first relevant result (MPOS) achieved for popular bookmark queries under each of the algorithms we proposed. The best combination remains SLLA+BMSR+SLAbS, with a gain of 37.00%. Thus, we conclude that for popular bookmark queries the combination of all methods is the best noise removal solution. Also, individually, Site Level Link Alliance (SLLA) produced the highest increase in PageRank quality.. After having evaluated our techniques on popular bookmark queries, we tested their performance over the randomly selected set. The MRR results for this scenario are displayed in Table 4 . Again, the best outcome was achieved when combining all the noise detection methods proposed, with an improvement of 20.92% in MRR, when compared to PageRank. Note that an improvement is harder to achieve under this setting, since the web pages searched are not necessarily popular, and thus many of them may have just a few in-going links and consequently a low PageRank score. Therefore, as removing links at the site level might also have the side effect of a further decrease of their PageRank score, they could become even more difficult to find. This is why both site level mutual reinforcement algorithms (BMSR and UMSR) resulted in a negative impact in the results, indicating that some site level mutual rein- Table 5 ). Individually, SLLA is still the best algorithm, whereas the best technique overall is again the combined SLLA+BMSR+SLAbS. Topic Queries. As mentioned earlier in this section, we evaluated topic queries using precision at the top 5 results (P@5) and at the top 10 results (P@10), as well as mean average precision (MAP). We first turn our attention to the experiment in which the output URLs assessed both as relevant and highly relevant are considered as good results. Table 6 presents the evaluation for the most popular 30 topic queries under this scenario. All results were tested for significance, and in both P@5 and P@10 no method manifested a significant gain or loss. Even so, in both P@5 and P@10 we see that BMSR has a slight gain over UMSR. SLLA exhibited the greatest gain in P@5, but the results were relatively similar for all algorithms in P@10. As for MAP, most of the results (except for SLAbS, BMSR, and their combination) had significant gain on MAP, when compared with the database without noise removal. Finally, SLAbS performance was rather poor. However, this behavior of SLAbS was recorded only with this kind of queries, where it is also explainable: Some very popular sites might indeed get an abnormal support from several of their fans; some would consider this as noise, but our testers apparently preferred to have the ranks of these sites boosted towards the top. The best individual method was SLLA and the best combination was SLLA with BMSR, which was better than the combination of all three methods due to the negative influence of SLAbS.. The same experiment was then performed for the 30 randomly selected topic queries. Its results are depicted in Table 7 due to this special behavior of our algorithms, under this setting the main contributor to the combined measures was SLAbS, thus yielding the best MAP score for BMSR+SLAbS. Before concluding this analysis, we also measured the quality of our methods under the same setting, but considering only the highly relevant output URLs as good results (recall that our subjects evaluated each URL as irrelevant, relevant and highly relevant for each query). For the popular topic queries (Table 8) , the performance of the individual methods was similar to the scenario that considered both relevant and highly relevant results, with the main difference being that here SLAbS gains about 12% over the database without noise removal, instead of losing. This is because the sites previously discovered due to noise (i.e., those being very popular, but also abnormally supported by some fans) were considered only relevant by our testers, and thus not included in this more strict experiment. Finally, for the randomly selected queries (Table 9) , SLAbS again showed the best individual performance (just as in the sibling experiment considering both kinds of relevance judgments), with the overall top scores being achieved for SLLA+BMSR+SLAbS and BMSR+SLAbS.. Conclusion. In order to make our results more clear, we also plotted their relative gain over regular PageRank (i.e., without noise removal). Figure 4 depicts this gain in percentage for bookmark queries and Figure 5 depicts it for topic queries. We first note that UMSR yielded negative results in three of the four experiments with bookmark queries, which makes it less preferable to its sibling BMSR, even though it performed better than the latter with topical queries. Also, we observe that SLAbS performed quite well under both broad experimental settings, but SLLA is clearly the best single approach for bookmark queries. Finally, all combined measures performed very well, with SLLA+BMSR+SLAbS being the best combination. Most of the previous works on noise are focused on solving the problem of detecting spam, instead of detecting noise in a general sense. Good solutions for the spam detection problem are important and difficult because they need to deal with adversaries that continuously try to deceive search engine algorithms. As the search output is usually ordered using a combination of the various algorithms available to assess the quality of each result (e.g., PageRank, HITS, TFxIDF, etc.), spammers have devised specific schemes to circumvent each of these measures. Consequently, the search engines responded with detection or neutralization techniques. This caused the spammers to seek new rank boosting methods, and so on. Since our work here is focused on identifying noise on the link structure of web collections, we will present in this section only the most recent anti-spam techniques for link-based algorithms.. Ranking Based Approaches. The currently known types of artificial noisy link structures which could boost the rank of one or more web pages have been investigated by Gyögyi and GarciaMolina. [16] . They manually build toy-scale link farms (networks of pages densely connected to each other) or alliances of farms and calculate their impact upon the final rankings. We used their results to design some of our spam fighting algorithms.. The seminal article of Bharat and Henzinger [7] has indirectly addressed the problem of noise detection on the web. Though inherently different from our approach, it is the only work we found in the literature that detects noise at a site level. Further, the authors provide a valuable insight into web noise link detection: They discovered the existence of "mutually reinforcing relationships" and proposed to assign each edge an authority weight of 1/k if there are k pages from the one site pointing a single document from another site, as well as a hub weight of 1/l if a page from the first site is pointing to l documents residing all on the second site. Authors use this information to change HITS [21] , a specific link analysis algorithm. We believe their solution can be used to complement our approach, and we intend to adapt it and integrate it into our algorithms in future work. Later, Li et al. [24] also proposed an improved HITS algorithm to avoid its vulnerability to small-in-largeout situations, in which one page has only a few in-links but many out-links. Nevertheless, their work only focuses on this specific problem, thus not tackling noise detection at all.. Another important work is SALSA [23] , where the "Tightly-Knit (TKC) Community Effect" is first discussed. The organization of pages into such a densely linked graph usually results in increasing their scores. The authors proposed a new link analysis algorithm which adopts two Markov chains for traversing the web graph, one converging to the weighted in-degree of each page, for authority scores, and the other converging to its weighted out-degree, for hub scores. The approach resembles popularity ranking, which was also investigated by Chakrabarti [12] and Borodin et al. [8] . However, it does not incorporate any iterative reinforcement and is still vulnerable to some forms of the TKC effect [26] .. Zhang et al. [29] discovered that colluding users amplify their PageRank score with a value proportional to Out(1/ ), where is the damping factor. Thus, they propose to calculate PageRank with a different for each page p, automatically generated as a function of the correlation coefficient between 1/ and P ageRank(p) under different values for . Their work is extended by Baeza-Yates et al. [3] , who study how the PageRank increases under various collusion (i.e., nepotistic) topologies and prove this increase to be bounded by a value depending on the original PageRank of the colluding set and on the damping factor.. BadRank [2, 27] is one of the techniques supposed to be used by search engines against link farms. It is practically an inverse PageRank, in which a page will get a high score if it points to many pages with high BadRank, as depicted in the formula below:. The exact expression of IB(p) is not known, but it represents the initial BadRank value of page p as assigned by spam filters, etc. The algorithm is complementary to our approaches. We therefore plan to investigate in a further work the idea of propagating the badness score of a page as an extension on top of the algorithms presented in this paper. TrustRank [18] proposes a rather similar approach, but focused on the good pages: In the first step, a set of high quality pages is selected and assigned a high trust; then, a biased version of PageRank is used to propagate these trust values along out-links throughout the entire web. The algorithm is orthogonal to our approaches: Instead of seeking for good pages, we attempt to automatically identify and penalize malicious links, thus decreasing the authority of the bad pages they point to. This ensures that good pages that are accidentally part of a malicious structure will be at most downgraded, but never dismissed from the possible set of query results (as in TrustRank).. SpamRank [5] resembles an "opposite TrustRank": First, each page receives a penalty score proportional to the irregularity of the distribution of PageRank scores for its in-linking pages; then, Personalized PageRank is used to propagate the penalties in the graph. The advantage over TrustRank is that good pages cannot be marked as spam, and comes at a cost of higher time complexity. Our approach is similar with respect to penalizing bad pages, but we build our set of malicious candidates much faster, by identifying abnormal link structures, instead of analyzing the distribution of PageRank scores for the in-linking pages of each page.. Finally, Wu and Davison [27] first mark a set of pages as bad, if the domains of n of their out-links match the domains of n of their in-links (i.e., they count the number of domains that link to and are linked by that page). Then, they extend this set with all pages pointing to at least m pages in the former set, and remove all links between pages marked as bad. Finally, new rankings are computed using the "cleaned" transition probability matrix. Their algorithm is complementary to our approach, as it operates at the lower level of web pages, instead of sites. In [28] , the same authors build bipartite graphs of documents and their "complete hyperlinks" 1 to find link farms of pages sharing both anchor text and link targets (i.e., possibly automatically created duplicate links).. Other Approaches. While most of the web noise detection research has concentrated directly on the link analysis algorithms used within current search engines, another significant stream of activity was dedicated to designing innovative third party solutions to detect such unwanted hyperlinks. Kumar et al. [22] used bipartite graphs to identify web communities and marked as nepotistic those communities having several fans (i.e., pages contributing to the core of the bipartite graph with their out-links) residing on the same site. Roberts and Rosenthal [26] analyze the number of web clusters pointing to each target page in order to decrease the influence of TKCs. They propose several methods to approximate these clusters, but they evaluate their approach only on a minimal set of queries. A rather different technique is employed in [1] , where the authors present a decision-rule classifier that employs 16 connectivity features (e.g., average level of page in the site tree, etc.) to detect web site functionality. They claim to have successfully used it to identify link spam rings as well, but no details are given about the importance of each feature for accomplishing this task. 1 Hyperlinks having the anchor text attached to them.. Chakrabarti [11] proposed a finer grained model of the web, in which pages are represented by their Document Object Models, with the resulted DOM trees being interconnected by regular hyperlinks. The method is able to counter "nepotistic clique attacks", but needs more input data than our algorithms (which are based exclusively on link analysis). Also, since we specifically target noise removal, we are able to identify different types of link anomalies.. Fetterly et al. [15] use statistical measures to identify potential spam pages. Most of the features they analyze can be modeled by well known distributions, thus placing outliers in the position of potential spammers. After a manual inspection, the vast majority of them seemed to be spammers indeed. A related technique to detect spam pages is based on machine learning algorithms: Davison [13] uses them on several features of URLs (e.g., similar titles, domains, etc.) in order to identify nepotistic links on the web.. Before proceeding to discuss our algorithms, we should note that quite several other types of link noise exist besides spam. The most common is caused by mirror hosts and can be eliminated using algorithms such as those proposed by Broder et al. [10] or Bharat et al. [6] . Also, navigational links are intended to facilitate web browsing, rather than expressing true votes of trust. One work indirectly related to this type of links is [14] , where the authors defined web documents as a "cohesive presentation of thought on a unifying subject" and proposed using these entities for information retrieval, instead of the regular web pages. Their work is however orthogonal to ours, as they seek to identify the correct web entities, whereas we propose solutions to remove noisy links from search engine databases. Preliminaries. Link analysis algorithms are founded on the representation of the web as a graph. Throughout the paper we will refer to this graph as G = (V, E), where V is the set of all web pages and E is a set of directed edges < p, q >. E contains an edge < p, q > iff a page p links to a page q. In(p) represents the set of pages pointing to p (the in-neighbors of p) and Out(p) the set of pages pointed to by p (the out-neighbors of p). We denote the p-th component of a vector v as v(p). Finally, let A be the normalized adjacency matrix corresponding to G with, Aij = 1 |Out(j)| if page j links to page i and Aij = 0 otherwise.. We now take a closer look at the PageRank algorithm. Since it is one of the most popular link analysis algorithms, we chose to use it to evaluate the impact of our noise removal methods.. PageRank [25] computes web page reputation scores starting from the idea that "a page should receive a high rank if the sum of the ranks of its in-neighbors is also high". Given a page p, the PageRank formula is:. The damping factor c < 1 (usually 0.15) is necessary to guarantee convergence and to limit the effect of rank sinks [9] . We evaluated the impact of our noise removal techniques on the link database of the TodoBR search engine. This database consisted of a collection of 12,020,513 pages extracted from the Brazilian web, connected by 139,402,245 links. As it represents a considerably connected snapshot of the Brazilian web community, which is probably as diverse in content and link structure as the entire web, we think it makes a realistic testbed for our experiments.. In order to evaluate the impact of our algorithms within practical situations, we extracted test queries from the TodoBR log, which is composed of 11,246,351 queries previously submitted to the search engine. We divided these selected queries in two groups:. 1. Bookmark queries, in which a specific web page is sought; 2. Topic queries, in which people are looking for information on a given topic, instead of some page. Each query set was further divided in two subsets, as follows:. • Popular queries: Here, we selected the top most popular bookmark / topic queries found in the TodoBR log. These queries usually search for well known web sites and are useful to check what happens to these most commonly searched pages after the noise removal methods have been applied.. • Randomly selected queries: Here, we selected the queries randomly. These queries tend to search for less popular sites and show the impact of our techniques on pages that are probably not highly ranked by PageRank. Then, 14 undergraduate and graduate computer science students (within different areas) evaluated the selected queries under various experimental settings. All of them were familiar with the Brazilian web pages and sites, in order to ensure more reliability to our experiments.. The bookmark query sets contained each 50 queries, extracted using the above mentioned techniques. All bookmark query results were evaluated using MRR (Mean Reciprocal Ranking), which is the metric adopted for bookmark queries on the TREC Conference 2 and is computed by the following equation:. where QS is the set of queries we experiment on, and P osRelAns(qi) is the position of the first relevant answer in the rankings output for query qi. MRR is the most common metric for evaluating the quality of results in bookmark queries. As it can be seen, its formula prioritizes methods that obtain results closer to the top of the ranking, adopting an exponential reduction in the scores (i.e., higher scores are better), as the position of the first relevant answer in the ranking increases. Also, MRR is very good at assessing the "real life" performance of the search engine, as the URLs most likely to be visited are those returned at the very top of the result list [20] . However, MRR is not sensible to pages having huge drops in position (e.g., from place 15 to place 40). Therefore, we also adopted another measure, mean position, (denoted MPOS in the tables to follow), which computes the average position of the first relevant answer in the output provided for each query. This metric results in a linear increase in the scores (higher is worse) as the position of the relevant answer increases.. For topic queries, we used two sets of 30 queries also selected from the TodoBR log as described previously. These different queries evaluate the impact of our noise removal algorithms when searching for some given topics. In this case, we evaluated the 2 http://trec.nist.gov/ results using the same pooling method as used within the Web Collection of TREC [19] . We thus constructed query pools containing the first top 20 answers for each query and algorithm. Then, we assessed our output in terms of various precision based metrics. The precision of an algorithm is defined as the number of relevant results returned divided by the total number of results returned. For each algorithm, we evaluated the Mean Average Precision (MAP), the precision at the first 5 positions of the resulted ranking (P@5), as well as the precision at the top 10 output rankings (P@10). In all cases the relevant results were divided in two categories, (1) relevant and (2) highly relevant. Also, we processed all queries according to the user specifications, as extracted from the TodoBR log: phrases, Boolean conjunctive or Boolean disjunctive. The set of documents achieved for each query was then ranked according to the PageRank algorithm, with and without each of our link removal techniques applied. Finally, all our results were tested for statistical significance using T-tests (i.e., we tested whether the improvement over PageRank without any links removed is statistically significant).. In all the forthcoming tables, we will label the algorithms we evaluated as follows:. • ALL LINKS: No noise removal.. • UMSR: Uni-directional Mutual Site Reinforcement.. • BMSR: Bi-directional Mutual Site Reinforcement.. • SLAbS: Site Level Abnormal Support.. • SLLA: Site Level Link Alliances.. • Combinations of the above, in which every method is applied independently to remove (UMSR, BMSR, SLAbS) or downgrade (SLLA) links, and then PageRank is applied on the resulting "cleaned" graph. Several combinations have been tested, but due to space limitations, only the best will be presented here. Algorithm specific aspects. Another important setup detail is how to divide the collection into web sites, as the concept of web site is rather imprecise. In our implementation, we adopted the host name part of the URLs as the keys for identifying individual web sites. This is a simple, yet very effective heuristic to identify sites, as pages with different host names usually belong to different sites, while those with identical host names usually belong to the same site.. As UMSR, BMSR and SLAbS all use thresholds to determine whether links between pairs of sites are noisy or not, it is important to tune such thresholds in order to adjust the algorithms to the collection in which they are applied. For the experiments we performed, we adopted the MRR results achieved for bookmark queries as the main parameter to select the best threshold. This metric was adopted because link information tends to have a greater impact on bookmark queries than on topic queries. Further, MRR can be calculated automatically, reducing the cost for tuning. The best parameters for each method depend on the database, the amount of noisy information and the requirements of the search engine where they will be applied. Table 1 presents the best thresholds we found for each algorithm using MRR as the tuning criteria. These parameters were adopted in all the experiments presented.

O O B O O O O O O O B I O O O O B O O O O O O O B O B O O O O O O B O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O B O O O O B O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O B O O O O O O B O O O B I O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B I O O O O O O B O O O O O O O O O O B O B O O O O O B O O O O O O O B O O O O B I O O O O O O O B O O O O O O O O O O O O O O O B O O B B O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B I O O O O O O O O O O O O O B O O O O O O O B O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B I O O O O O O O O O O B O O B O O O O O O O O O O O B O O O O O B O O O O O O O B O O O B O O O B O O O O O O O O O B O O B O O O O O O B O O O O O B O O O O O O O O B O O O O O O O O O O O O O O B O O O O O B O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B I B O O O O O O B O O O B O O O O O O O O O B I B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O B I O O O O O O B O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B I O O O O B O B I O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O B I O O O O O O O O B O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O B O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B I B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B B O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O B O O B O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O B O B I O O B I O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B B O O O O O O O O B I O O O O O O O O O O O O B O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O B I O B O O O O O O O O O O O B O O O B O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B I O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O