Optimal Randomized RANSAC Abstract-A randomized model verification strategy for RANSAC is presented. The proposed method finds, like RANSAC, a solution that is optimal with user-specified probability. The solution is found in time that is close to the shortest possible and superior to any deterministic verification strategy. A provably fastest model verification strategy is designed for the (theoretical) situation when the contamination of data by outliers is known. In this case, the algorithm is the fastest possible (on the average) of all randomized RANSAC algorithms guaranteeing a confidence in the solution. The derivation of the optimality property is based on Wald's theory of sequential decision making, in particular, a modified sequential probability ratio test (SPRT). Next, the R-RANSAC with SPRT algorithm is introduced. The algorithm removes the requirement for a priori knowledge of the fraction of outliers and estimates the quantity online. We show experimentally that on standard test data, the method has performance close to the theoretically optimal and is 2 to 10 times faster than standard RANSAC and is up to four times faster than previously published methods. T HE RANdom SAmple Consensus (RANSAC) algorithm introduced by Fishler and Bolles in 1981 [6] is a widely used robust estimator that has become a standard in the field of computer vision [1] . RANSAC and related hypothesize-and-verify methods [4] , [11] , [12] , [13] , [14] , [15] have been applied to many vision problems.. The RANSAC algorithm proceeds as follows: Repeatedly, subsets of the input data (for example, a set of tentative correspondences) are randomly selected (with replacement), and model parameters fitting these subsets are computed. In the second step, the quality of the parameters is evaluated on the input data. Different cost functions have been proposed [15] , the standard being the number of data points (inliers) consistent with the model. The process is terminated when the probability of finding a better model becomes lower than a user-specified probability 0 . The 1 À 0 confidence in the solution holds for all levels of contamination of the input data, that is, for any number of outliers within the input data. The speed of standard RANSAC depends on two factors: The number of random samples and the number N of the input data points. In all common settings where RANSAC is applied, almost all models whose quality is verified are incorrect with arbitrary parameters originating from contaminated samples. Such models are consistent with only a small number of the data points. In [9] , Matas and Chum showed how this property could be exploited to increase the speed of RANSAC. The algorithm, called R-RANSAC, reduces the time needed for the model evaluation step by introducing a two-stage procedure. First, a statistical test is performed on d randomly selected data points ðd ( NÞ. Evaluation of the remaining N À d data points is carried out only if the first d data points are inliers. The speed up of R-RANSAC depends on the probabilities of the two types of errors committed in the pretest, the rejection of an uncontaminated model and the acceptance of a contaminated model. This idea was modified by Nistér to include competitive verification of models. The algorithm performed impressively in a real-time structure from motion system [12] . The main limitation of Nistér's preemptive RANSAC is that a fixed number of models are evaluated, which is equivalent to an a priori assumption that a lower bound on the fraction of inliers is known. This limits the applicability of preemptive RANSAC in problems where the fraction of inliers varies widely, such as wide baseline stereo.. As noted in [9] , the two-stage procedure of R-RANSAC is not optimal. As a main contribution of this paper, we define an optimal hypothesis evaluation procedure, that is, a method for randomized model quality evaluation that returns, in the fastest average time possible, a solution with the confidence 1 À 0 . The derivation of the optimality property is based on Wald's theory of sequential decision making [16] .. In Section 2, the concept of randomization of the verification step is described in detail. Section 3 reviews previous work relevant to the topic. In Section 4, we introduce the relevant parts of Wald's decision theory and show how its results can be brought to bear on the problem of minimizing RANSAC runtime. The RANSAC with SPRT algorithm [10] is described in detail in Section 5. In Section 6, the theoretical results are experimentally verified on standard stereo matching problems. The paper is concluded in Section 8. An optimal sequential strategy for randomized evaluation of model quality in RANSAC was derived. A method for the estimation of two probabilities characterizing the problem and critically influencing the design of the optimal strategy was proposed and experimentally verified. A termination criterion derived in the paper guarantees that the solution found is correct with confidence 1 À 0 , as in standard RANSAC.. The properties of R-RANSAC with SPRT were tested on a diverse set of standard data. The test problems included homography estimation and epipolar geometry estimation in both wide and narrow baseline setting.. R-RANSAC with SPRT was 2 to 9 times faster than standard RANSAC and up to 3.5 times faster than R-RANSAC and was outperformed on more challenging tasks. Experiments on a large set of synthetic problems confirms the quasi-optimality of R-RANSAC with SPRT. . For more information on this or any other computing topic, please visit our Digital Library at www.computer.org/publications/dlib. The idea of randomized verification of hypotheses in RANSAC appeared only recently [3] , and the related literature is thus limited. We review the original R-RANSAC that relied on the T ðd;dÞ test [3] , [9] and the two algorithms it inspired: the preemtive RANSAC [12] and the RANSAC with a bail-out test [2] . . Here, stands for the probability that a bad sample passes the preverification test. Note that it is important that ( 1 À , so that a bad (contaminated) sample is consistent with a smaller number of data points than a good sample. In [3] , the optimal value for d is derived in detail. The constants introduced in the previous section are expressed as. where is the probability that a data point is consistent with an arbitrary model. For a broad range of values of " and , the optimal value of d is d ¼ 1. Therefore, without any prior knowledge of " and , the suggested test (from the T d;d class of tests) is T 1;1 .. Note that instead of drawing m data points and verifying only d data points in the test, the following approach can be adopted in the T d;d test. Draw m þ d data points randomly, and fit model parameters to the sample. Measure the error on the m þ d data points in the sample. If the error is smaller than a predefined threshold Á, that is, there is a model that fits all the data well, proceed with the verification step; otherwise, generate a new hypothesis. The advantage of such an approach is that a single model can be obtained from m þ d data points (in contrary to m S models from m data points only), and also a model of higher accuracy is typically obtained if more points are used. This is specific to the T d;d class of tests, where the consistency of all d data points is required.. 1. In the definition of "good" and "bad" models, we ignored the fact that for certain models, for example, the fundamental matrix, the optimal model can be obtained from a sample contaminated by an outlier that is by coincidence consistent with the correct epipolar geometry. We also ignored the fact that an uncontaminated, all-inlier sample may not lead to a "good" model due to noise and poor conditioning of the estimation process. The problem can be overcome with so-called local optimization, for details, see [4] . Neither simplification has an impact on the derivations below.. Despite the simplicity of T 1;1 test, experiments have shown that preverification can reduce RANSAC runtime significantly [9] . Several experiments were performed comparing the proposed R-RANSAC with SPRT to three other RANSAC algorithms:. 1. standard RANSAC that verifies all correspondences for every model, 2. R-RANSAC with the T d;d test [9] that rejects the model after the first checked correspondence is inconsistent with it ðd ¼ 1Þ, 3. R-RANSAC with the a priori SPRT, that is, the R-RANSAC with SPRT designed for the true values of " and (labeled SPRT Ã ), and 4. RANSAC with the bail-out test proposed by Capel [2] . The results achieved with an a priori SPRT show the best achievable performance of RANSAC with a randomized 7 . RANSAC verifies, on the average, À logð 0 Þ "good" models. For the typical 0 ¼ 0:05, a "good" model is hypothesized three times prior to the termination of the algorithm.. verification step for a problem characterized by a given and ".. For epipolar geometry estimation, the time needed to compute model parameters t M ¼ 200 was set within the range observed in a large number of experiments (that is, in our implementation, checking whether a correspondence is consistent with a fundamental matrix is 200 times faster than estimating the matrix). The exact value depends on many factors including the CPU speed and type. The constant m S ¼ 2:38 was set to the experimentally observed average of the number of models generated by the 7-point algorithm per sample [5] . 8 The initial values of and " were set to 0 ¼ 0:05 and " ¼ 0:2, respectively.. For homography estimation, the values were set as follows: t M ¼ 200, m S ¼ 1, 0 ¼ 0:01, and " 0 ¼ 0:1.

O O B O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Analysis of colour constancy algorithms using the knowledge of variation of correlated colour temperature of daylight with solar elevation In this article, we present an investigation of possible improvement of the colour constant reflectance features that can be obtained from daylight illuminated scenes using pixel-level colour constancy algorithms (model-based algorithm: S Ratnasingam, S Collins, J. Opt. Soc. Am. A 27, 286-294 (2010) and Projection-based algorithm: GD Finlayson, MS Drew, IEEE ICCV, 2001, pp. 473-480). Based on the investigation we describe a method to improve the performance of the colour constancy algorithms using the correlation between the correlated colour temperature of measured daylight with the solar elevation and phase of the day (morning, midday and evening). From this observation, the data from 1 year are used to create a solar elevation and phase of day-dependant method of interpreting the information obtained the colour constancy algorithms. Test results show that using the proposed method with 40-dB signal-to-noise ratio the performance of the projection-based algorithm and modelbased algorithm can be improved on average by 33.7 and 45.4%, respectively. More importantly, a larger improvement (85.9 and 113.7%) was obtained during the middle period of each day which is defined as when the solar elevation is larger than 20°. Colour is a useful feature in several machine vision applications including object recognition and image indexing. However, the apparent colour of an object varies depending on the viewing environment, particularly in scenes illuminated by daylight. Colour-based recognition in naturally illuminated scenes is therefore a difficult problem and existing techniques are not effective in real scenes [1] [2] [3] [4] [5] [6] . The problems with obtaining colour information from naturally illuminated outdoor scenes arise from the uncontrolled changes to both the intensity and the spectral power distribution of daylight. The resulting variations in the recorded colour of the same object under different daylight conditions make it difficult to use colour as a reliable source of information in machine vision applications [3] [4] [5] [6] .. In contrast to camera-based systems, the human visual system's perception of the colour of an object is largely independent of the illuminant. Several approaches to replicating the colour constancy achieved by the human visual system have been proposed [6] . However, many of these algorithms assume that scene is illuminated uniformly and shadows mean that this assumption is not always valid for outdoor scenes. The assumption that a scene is uniformly illuminated is avoided by those colour constancy algorithms that only use pixel-level information. Marchant and Onyango [7] , Finlayson and Hordley [8] and Romero et al. [9] have all proposed methods for obtaining a single illuminant invariant feature from pixel data under daylight based on the assumptions that the spectral responses of the sensors are infinitely narrow and that the power spectral density of the illuminant can be approximated by the power spectral density of a blackbody illuminant. However, a single feature potentially leads to confusion between some perceptually different colours [6] . This confusion can be avoided using the method proposed by Finlayson and Drew [10] that obtains two illuminant independent features from the responses of four sensors with different spectral responses. Based on the same assumptions of infinitely narrow band sensors and the blackbody model of the illuminant, Ratnasingam and Collins [11] proposed a simple method of obtaining two illuminant independent features from the responses of four sensors. Since neither of the assumptions upon which all these methods is strictly applicable, Ratnasingam and Collins [11] also proposed a method for determining the quality of the spectral information that can be obtained from the features obtained using these methods. One important conclusion from their work was that pixel-level colour constancy algorithms extract useful information from the responses of sensors whose spectral response covers a wavelength range of 80 nm or less [11] . Subsequently, Ratnasingam et al. [12] investigated the possibility of optimising the spectral responses of the image sensors to improve the quality of the illuminant invariant features that can be obtained from daylight illuminated scenes using pixellevel colour constancy algorithms. The conclusion of this study was that the widely available International Commission on Illumination (CIE) standard illuminants was not representative enough to the actual daylight measurements obtained under widely varying weather conditions to be used to reliably optimise the spectral responses [12] .. Studying the pattern of variation of the daylight spectra would be very useful because daylight dominates in the day time and some important machine vision applications, including remote sensing, controlling unmanned vehicles and simultaneous localisation and mapping, are based upon processing outdoor scenes. The variation in spectral power distribution of the daylight due to the variation in solar elevation, time of year and geographic location leads to difficulties in reliably recognising objects or scenes. In this article, we investigate the advantages of using solar elevation and phase of the day as auxiliary information to enhance the performance of a colour constancy algorithm. This auxiliary information can be calculated if the time of day, date and the geographic location of the camera are known. However, it is difficult to model such a relation with the artificial illuminants. But, one could use the knowledge of whether the scene is indoor or outdoor to account for the illuminant in these scenarios. Based on this evidence an approach for improving the performance of colour constancy algorithms that can be obtained from daylight illuminated scenes is proposed based upon the observed variation of the correlated colour temperature (CCT) of measured daylight with solar elevation and date. Here, it is worth noting that in our investigation for possible improvement of colour constancy algorithms we only use the approximate geographic location (example country or region) and phase of the day (i.e. morning, midday and afternoon) as auxiliary information to the colour constancy algorithms. The performance improvement obtained using the proposed approach has been illustrated using two existing colour constancy algorithms proposed by Finlayson and Drew [10] and that proposed by Ratnasingam and Collins [11] . Here, we should note that the algorithms chosen to investigate the performance enhancement of colour constancy using solar elevation and date, do not rely on the content of the rest of the scene in extracting the illuminant invariant features. This choice was made because outdoor scene might consist of regions illuminated by direct sunlight or skylight or under shadow and the power spectrum of the incident light in these regions will not be the same [13] . This means in the outdoor scenes where the intensity and the relative power spectrum vary spatially, the performance of any algorithm that processes the entire scene to obtain an estimate for the illuminant, significantly degrades in any real-world scenes therefore not appropriate for machine vision applications. For this reason, we have chosen two algorithms that extract illuminant invariant features at pixel level rather than processing the whole scene.. The remainder of this article is organised as follows: In Section 2, a brief description of the algorithm of obtaining useful features related to the reflectance of a surface proposed by Finlayson and Drew [10] and Ratnasingam and Collins [11] is presented. In Section 3, a description of the proposed method for improving the degree of illuminant invariance obtained from the features determined using the two different methods is given. Simulation results to assess the effectiveness of the proposed method are then presented in Section 4. Finally, Section 5 contains the conclusions of this study. If the spectrum of the scene illuminant can be approximated by the spectrum of a blackbody then pixel-based colour constancy algorithms can be used to extract features that are almost independent of the illuminant. The blackbody approximation used to derive these algorithms is most appropriate when the illuminant is daylight. Results have been presented using measured daylight spectra that show that the residual illuminant dependence of the features is caused by a combination of noise, finite spectral width and the non-blackbody spectrum. The latter effect means that the features obtained from a reflective surface vary with the spectrum of the daylight, usually characterised by its CCT. An investigation of the CCTs of daylight spectra measured in Granada, Spain, over 2 years shows that the CCT of daylight varies with the solar elevation and to a lesser extent on the date. We investigated the optimum number of periods in a year and the solar elevation threshold that results the best performance. Based upon these observations, the measured daylight spectra for 1 year have been used to create 12 sets of illuminants. These illuminants have then been used to create boundaries in the feature space that can be used to distinguish between pairs of surfaces whose colours are very good matches to each other. Using the data from a second year which is different from the training set, it has been shown that for good quality images the ability to successfully distinguish these surfaces is significantly improved on average using the phase of the day and date information by 33.7 and 45.4% for projection-based and model-based algorithms, respectively. In particular, a larger improvement (average improvement: 85.9 and 113.7%) was obtained during the middle period of each day which is defined as when the solar elevation is larger than 20°. This method would therefore seem to be particularly useful for scenes illuminated by daylight in the middle of the day. One possible application that suits these restrictions is precision farming, in which machine vision techniques can be used to target the treatment of small areas of a field or individual plants [22, 23] . In the future, the possibility of optimising the sensor characteristic and using a third illuminant invariant feature will be investigated.

O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Online Reinforcement Learning for Dynamic Multimedia Systems In our previous work, we proposed a systematic cross-layer framework for dynamic multimedia systems, which allows each layer to make autonomous and foresighted decisions that maximize the system's long-term performance, while meeting the application's real-time delay constraints. The proposed solution solved the cross-layer optimization offline, under the assumption that the multimedia system's probabilistic dynamics were known a priori, by modeling the system as a layered Markov decision process. In practice, however, these dynamics are unknown a priori and therefore must be learned online. In this paper, we address this problem by allowing the multimedia system layers to learn, through repeated interactions with each other, to autonomously optimize the system's long-term performance at run-time. The two key challenges in this multi-agent (layered) learning setting are: (i) each layer's learning performance is directly impacted by not only its own dynamics, but also by the learning processes of the other layers with which it interacts; and, (ii) selecting a learning model that appropriately balances time-complexity (i.e. learning speed) with the multimedia system's limited memory and the multimedia application's real-time delay constraints. We propose two reinforcement learning algorithms for optimizing the system under different design constraints: the first algorithm solves the cross-layer optimization in a centralized manner, and the second solves it in a decentralized manner. We analyze both algorithms in terms of their required computation, memory, and inter-layer communication overheads.After noting that the proposed reinforcement learning algorithms learn too slowly, we introduce a complementary accelerated learning algorithm that exploits partial knowledge about the system's dynamics in order to dramatically improve the system's performance. In our experiments, we demonstrate that decentralized learning can perform as well as centralized learning, while enabling the layers to act autonomously. Additionally, we show that existing application-independent reinforcement learning algorithms, and existing myopic learning algorithms deployed in multimedia systems, perform significantly worse than our proposed application-aware and foresighted learning methods.State-of-the-art multimedia technology is poised to enable widespread proliferation of a variety of life-enhancing applications, such as video conferencing, emergency services, surveillance, telemedicine, remote teaching and training, augmented reality, and distributed gaming. However, efficiently designing Online Reinforcement Learning for Dynamic Multimedia SystemsNicholas Mastronarde*, Mihaela van der Schaar 2/35 and implementing such delay-sensitive multimedia applications on resource-constrained, heterogeneous devices and systems is challenging due to the real-time constraints, high workload complexity, and the time-varying environmental dynamics experienced by the system (e.g. video source characteristics, user requirements, workload characteristics, number of running applications, memory/cache behavior etc.).Cross-layer adaptation is an increasingly popular solution for addressing these challenges in dynamic multimedia systems (DMSs) [1]-[8]. This is because the performance of DMSs (e.g. video rate-distortion costs, delay, and power consumption) can be significantly improved by jointly optimizing parameters, configurations, and algorithms across two or more system layers (i.e. the application, operating system, and hardware layers), rather than optimizing and adapting them in isolation. However, existing crosslayer solutions have several important limitations.The majority of these solutions require a central optimizer to coordinate the application, operating system, and hardware adaptations to optimize the performance of one or multiple multimedia applications sharing the DMS's limited resources. Unfortunately, these approaches ignore the fact that the majority of modern system's are comprised of layers (components or modules) that are designed by different manufacturers, which makes such tightly integrated designs impractical, if not impossible [8] [18] [19] .Another limitation of many existing cross-layer solutions for DMSs is that they are myopic. In other words, cross-layer decisions are made reactively in order to optimize the immediate utility, without considering the impact of these decisions on the future utility. However, in DMSs, it is essential to predict the impact of the current decisions on the long-term utility because the however, with the exception of our prior work [9], they have never been used for cross-layer DMS optimization, where all the layers make foresighted decisions.Single-layer learning solutions: The various multimedia system layers must be able to adapt at runtime to their experienced dynamics. Most existing learning solutions for multimedia systems are concerned with modeling the unknown and potentially time-varying environment experienced by a single layer. We refer to these as single-agent learning solutions (in our setting, an agent corresponds to a layer). In summary, while significant contributions have been made to enhance the performance of DMSs using cross-layer design techniques, no systematic cross-layer framework exists that explicitly considers:(i) the design and implementation constraints imposed by a layered DMS architecture; (ii) the ability of the various layers to autonomously make foresighted decisions in order to jointly maximize the DMS's utility; and, most importantly, (iii) how layers can learn their unknown environmental dynamics and determine how they impact and are impacted by the other layers.In this paper, similar to [6] [7] [8], we consider a multimedia system in which (i) a video encoder at the application layer makes rate-distortion-complexity tradeoffs by adapting its configuration and (ii) the operating system layer makes energy-delay tradeoffs by adapting the hardware layer's operating frequency. Our contributions are as follows:• We propose a centralized reinforcement learning algorithm for online cross-layer DMS optimization based on the well-known Q-learning algorithm. This centralized solution can be easily implemented if a single manufacturer designs all of the layers in the multimedia system. • We propose a novel multi-agent (layered) Q-learning algorithm, which allows the multimedia system layers to autonomously learn their optimal foresighted policies online, through repeated interactions with each other. This decentralized solution can be implemented if the layers are designed by different manufacturers. We show experimentally that the proposed layered learning algorithm, which adheres to the layered system architecture, performs as well as a traditional centralized learning algorithm, which does not.• We propose a reinforcement learning technique, which is complementary to the aforementioned learning algorithms, that exploits partial a priori knowledge about the system's dynamics in order to accelerate the rate of learning and improve overall learning performance. Unlike existing reinforcement learning techniques [14] , [15] , which can only learn about previously visited stateaction pairs, the proposed algorithm exploits our partial knowledge about the system's dynamics 4/35 in order to learn about multiple state-action pairs even before they have been visited.The remainder of this paper is organized as follows. In Section II, we present the cross-layer problem formulation. In Section III, we describe the two layer multimedia system model. In Section IV, we propose a centralized and layered (decentralized) Q-learning algorithm for run-time cross-layer optimization. In Section V, we propose a technique to accelerate the rate of learning by exploiting partial a priori knowledge that we have about the system's dynamics. In Section VI, we present our experimental results and we conclude the paper in Section VII.In this section, we define the layered structure of the system under study and formulate the crosslayer system optimization problem as a Markov decision process (MDP). In Section III, we discuss in detail how the considered DMS problem can be modeled within the proposed cross-layer MDP framework.We model the layered system as a tuple , ,, whereis a set of L autonomous layers, which participate 1 in the cross-layer optimization. Each layer is indexedwith layer 1 corresponding to the lowest participating layer (e.g. the HW layer) and layer L corresponding to the highest participating layer (e.g. the APP layer).• A is the global action set2 , where l A is the l th layer's local action set (e.g.the APP layer's available source-coding parameter configurations and the OS layer's available commands for switching the CPU's operating frequency).• S is the global state set, where l S is the l th layer's local state set (e.g. the set of buffer states at the APP layer).• p is the joint transition probability function mapping the global state, global action, and global next-state to a value in [ ] 0,1 , i.e.[ ]• R is the expected reward function, which maps the global state and global action to a real number representing the system's expected reward, i.e.Within the proposed DMS framework, we are interested in trading off multiple objectives including application performance (characterized by delay and video distortion) and power consumption. Hence, we 1 We note that for a layer to "participate" in the cross-layer optimization it must be able to adapt one or more of its parameters, configurations, or algorithms (e.g. the APP layer can adapt its source coding parameters); alternatively, if a layer does not "participate", then it is omitted.A is the L -ary Cartesian product.assume that the reward function is of the form:where l J is the local cost at layer l , which is independent of the states and actions at the other layers (e.g. the application's distortion and hardware's power consumption); the utility gain ( , )L L g a s is related to the delay experienced by the application at layer L given its own state and action, and the states of the other system layers, which support it (see Section III.C for details); and, l ω weights the relative importance of the layer costs with the utility gain. In Section III, we formulate an illustrative DMS problem within this layered modeling framework.Unlike existing cross-layer optimization solutions, which focus on optimizing the myopic (i.e.immediate) utility, the goal in the proposed cross-layer framework is to find the optimal actions at each stage n ∈ » that maximize the discounted sum of future rewardswhere the parameter γ ( 0 1 γ ≤ < ) is the "discount factor," which defines the relative importance of present and future rewards, and 0 s is the initial state. We refer to the Markov decision policy : π * S A , which maximizes the discounted sum of future rewards from each initial state 0 ∈ S s , as the optimal foresighted policy. We use a discounted sum of rewards instead of, for example, the average reward, because: (i) typical video sources have temporally correlated statistics over short time intervals such that the future environmental dynamics cannot be easily predicted without error [7] , and therefore the system may benefit by weighting its immediate reward more heavily than future rewards; and, (ii) The multimedia session's lifetime is not known a priori (i.e. the session may end unexpectedly) and therefore rewards should be maximized sooner rather than later.Throughout this paper, we will find it convenient to work with the optimal action-value function 6/35In this section, using the framework introduced above, we model a system in which two layers participate in the cross-layer optimization (i.e.). In particular, the APP layer makes ratedistortion-complexity tradeoffs by adapting its configuration and the OS layer makes energy-delay tradeoffs by adapting the HW layer's operating frequency (using, for example, the platform independent open standard known as the Advanced Configuration and Power Interface [11] ). Due to the fact that the OS layer controls the HW layer, we combine them into a single decision making layer, which we call the joint OS/HW layer.As in [21], we model the video source as a sequence of video data units (for example, video macroblocks, groups of macroblocks, or pictures), which arrive at a constant rate into the application's pre-encoding buffer. Similar to [11], we assume that the system operates over discrete time slots.Additionally, we assume that the time slots have variable length, such that one data unit is encoded in each time slot. We interchangeably refer to the time slot during which the n th data unit is encoded as the n th "time slot" or "stage," where n ∈ » .For clarity, instead of specifying the layers by their indices, in this section we use the subscripts APP to represent quantities related to the APP layer ( 2 l = ) and OS to represent quantities related to the joint OS/HW layer's parameters ( 1 l = ). Fig. 1 illustrates the system model, which we describe in detail in the following subsections, and Table I The n th data unit is processed at the HW layer's current operating frequency n f , which is a member of the joint OS/HW layer's state set OS { :We let the joint OS/HW layer's cost represent the power dissipated at operating frequencyP ⋅ is the system's power-frequency function (see [5] [7] for example power-frequency functions).At stage n , the OS layer can issue a command in its action set OS OS { : } u u = ∈ A S to change the HW layer's operating frequency. Similar to [11], we assume that there is a non-deterministic delay associated with the operating frequency transition such thatwhere { } { } :n n F F n = ∈ » is the sequence of operating frequencies; n f and 1 n f + are the operating frequencies at stage n and 1 n + , respectively; and,is the probability of a successful operating frequency transition (with probability 1 β − the operating frequency remains the same). In Fig. 1, this transition is illustrated by a temporal dependency between the frequency command (action) and operating frequency (state) in the joint OS/HW layer. Note that, regardless of the OS layer's action n u , the n th data unit is processed at the operating frequency OS n f ∈ S at the power-cost OS ( ) n J f .The n th data unit can be classified as being one of z N types in the state set where { } { } :n n Z Z n = ∈ » is the sequence of data unit types, and n z and 1 n z + are the types of the data units that are encoded in time slots n and 1 n + , respectively. We assume that the choice of data unit type is governed by an algorithm similar to the one in [20] ; therefore, the probabilities in (6) depend on the desired ratio of I, P, and B data units (e.g. IBBP…), the source characteristics (i.e. the APP layer's environment illustrated in Fig. 1 ), and the condition used to decide when to code an I frame [20] .The n th data unit can be coded using any one of h N encoding parameter configurations in the APP layer's action set APP { : n n C z h , respectively. Note that we do not need to know the distributions of these random variables to learn the optimal state-value function and optimal policy at runtime (see Section IV).We penalize the APP layer's action APP n h ∈ A by employing the Lagrangian cost measure used in the H.264/AVC reference encoder for making rate-distortion optimal mode decisions: i.e. we define the APP layer's cost as APP ( , )n n n n n n n n rdn n n d z h is the n th data unit's encoded distortion; ( , ) n n n b z h is its encoded bit-rate; and , n n n n n n q q q t z h f N q qwhere ( , ) ( , , ) n n n n n n n n c z his the n th data unit's processing delay, which depends on its complexity ( , )n n n c z h and the processor's operating frequency n f (in Fig. 1 , the coupling between the operating frequency and processing delay is represented by the inter-layer dependency arrow);   x is the integer part of x ; and [ ] max{ , 0}Note that ( , )n n n c z h is an instance of the random variable ( , )Recall that the system operates over discrete time slots of variable length, such that one data unit is encoded in each time slot. Hence, in (7), the 1 − indicates that the n th data unit departs the pre-encoding buffer after the non-deterministic encoding delay ( , , ) ( , )/ n n n n n n T f z h C z h f = (seconds). Meanwhile, the number of data units that arrive in the pre-encoding buffer during the n th time slot,n n n n n n n n t z h f t z h f η = ⋅ , is an instance of the random variableBased on (7) and (9), the pre-encoding buffer's state transition can be modeled as a controllable Markov chain with transition probabilities Noting that the transitions of the data unit's type (see (6)) and the pre-encoding buffer's state (see (10)) are conditionally independent given the current data unit type n z , the APP layer's transition probability function can be expressed as ( )where APP APP ( , ) n n n z q = ∈ S s andn n n z qare the APP layer's state at time slot n and 1 n + , respectively. Meanwhile, the joint OS/HW layer's transition probability function can be expressed as (see (5)Because the state transitions at the APP layer and joint OS/HW layer are independent, the global transition probability function defined in Section II.A can be factored as where s and a are defined as in Table I .Recall the definition of the reward function in (1). Thus far, we have defined the costs at each layer, but we have not defined the utility gain APP 2 ( , ) n n g a s . As mentioned in Section II.A, the utility gain is closely related to the application's experienced queuing delay in the pre-encoding buffer. Conventionally, multimedia buffers are used to enforce a maximum tolerable processing [22] or transmission delay [21] and as long as the buffer does not overflow (i.e. the overflow constraint is not violated), there is no penalty. In our setting, however, the multimedia system's dynamics are unknown and non-stationary; therefore, we cannot guarantee that the buffer overflow constraints will be satisfied. In light of this, we design the utility gain to non-linearly reward the system for maintaining queuing delays less than the maximum tolerable delay, thereby protecting against overflows that may result from a sudden increase in encoding delay. Formally, we define the utility gain at stage n as11/35 which is near its maximum when the buffer is empty (corresponding to zero queuing delay) and is minimized when the buffer is full. Note that ( , , ) n n n n n t t f z h = , but we omit the arguments for brevity.In Appendix A, we discuss in detail why (14) is a good definition for the utility gain.Given the utility gain defined in (14), the APP layer's cost APP ( , ) n n n n n n n n n n n n n n n n rdIV. LEARNING THE OPTIMAL DECISION POLICYAs we mentioned before, the multimedia system's dynamics (i.e. ( , ) R s a and ( , ) p ′ s | s a ) are unknown and therefore the optimal action-value function Q * and the optimal policy π * must be learned online, based on experience. However, it remains to explain how we can learn the optimal policy online despite these unknown quantities. Let us consider the following two options:• Direct estimation and policy computation: One option is to directly estimate the reward and transition probability function using, for example, maximum likelihood estimation, and then to perform value iteration 3 [15] to determine the optimal policy; however, this solution is far too complex because each iteration of the algorithm has complexity ( )O S A and the number of iterations required to converge to the optimal policy is polynomial in the discount factor γ . Also note that storing the estimated transition probability function incurs a memory overhead of ( ) 2 O S A , which for a large number of states is unacceptably large (e.g. our experimental test-bed in Section VI uses 765 states and 15 actions, and therefore requires over 33 MBs of memory to directly store the transition probability function using 32 bit single-precision floating point numbers).• Offline policy computation and online policy interpolation (supervised learning): To avoid using the complex value iteration algorithm online, in [12] the authors propose a supervised learning approach involving an offline stage and an online stage. Offline, they quantize every unknown probability parameter into NS samples (more samples yield more accurate results). For each quantized probability bin (acting as a training sample), they compute a corresponding policy 12/35 (acting as a desired output). Then, online, they directly estimate the transition probability function and, based on this, interpolate the offline computed policies to determine the current policy. For M parameters, this learning algorithm requires Clearly, given the multimedia system's limited memory and the multimedia application's real-time delay constraints, neither of the above solutions are practical. Thus, in our resource-constrained crosslayer setting with many states and actions (and many unknown parameters), we adopt a model-free reinforcement learning solution, which can be used to learn the optimal action-value function Q * and optimal policy π * online, without directly estimating the reward and transition probability functions.Specifically, we adopt a low complexity algorithm called Q-learning, which also has limited memory requirements (see Table III in Section IV.D). In our cross-layer setting, the conventional Q-learning algorithm can be implemented by a central optimizer --located at either the APP layer, the joint OS/HW layer, or a separate middleware layer.Central to the conventional centralized Q-learning algorithm is a simple update step performed at the end of each time slot based on the experience tuple (ET) ( ) 1 , , , n n n n r + s a s :n n n n n n n r Q Q δ γn n n n n n n n Q Q α δ It is well known that if (i) the rewards and transition probability functions are stationary, (ii) all of the state-action pairs are visited infinitely often, and (iii) n α satisfies the stochastic approximation 3 Policy iteration or linear programming could also be used, but these solutions are also too complex. 4 A greedy action * a is one that maximizes the current estimate of the action-value function, i.e. Subsequently, the optimal policy can be found using (4).During the learning process, it is not entirely obvious what the best action is to take in each state. On the one hand, the optimal action-value function can be learned by randomly exploring the available actions in each state. Unfortunately, unguided randomized exploration cannot guarantee acceptable performance during the learning process. On the other hand, taking greedy actions, which exploit the available information in the action-value function ( , ) Q s a , can guarantee a certain level of performance.Unfortunately, exploiting what is already known about the system prevents the discovery of new, better, actions. To judiciously trade off exploration and exploitation, we use the so-called ε -greedy actionε -greedy action selection: With probability 1 ε − , take the greedy action that maximizes the actionvalue function, i.e.s a ; and, with probability ε , take an action randomly and uniformly over the action set.We writes to show that a is an ε -greedy joint-action. Note that, if the ε -greedy action is taken from the optimal state-value function Q * , then actions corresponding to the optimal policy will only be taken with probability 1 ε − ; therefore, if ε does not decay to 0, then optimal performance will not be achieved. Fig. 2 illustrates the information exchanges required to deploy the centralized Q-learning algorithm in a two layer multimedia system during one time slot. In Fig. 2 , the top and bottom blocks represent the APP layer and joint OS/HW layer, respectively. The center block represents the centralized optimizer, which selects both layers' actions and updates the system's global action-value function Q . As we mentioned before, the centralized optimizer may be located at either the APP layer, the joint OS/HW layer, or a separate middleware layer. To best highlight the information exchanges, we illustrate the centralized optimizer as a separate middleware layer. Although we do not explicitly indicate this in Fig. 2 , the ET ( ) , , , r ′ s a s is used by the block labeled "Update ( , ) Q s a ," which performs the update step defined in (17). Thus far, we have discussed how the traditional Q-learning algorithm can be implemented by a centralized optimizer. As we discussed in the introduction, this centralized solution is most appropriate when a single manufacturer designs all of the system's layers. However, we are also interested in developing learning solutions that allow layers designed by different manufactures to act autonomously, but still cooperatively optimize the system's performance. This is a challenging problem in our crosslayer setting because the dynamics experienced at each layer may depend on the learning processes of the other layers. Hence, a key challenge is determining how to coordinate the autonomous learning processes of the various layers to enable them to successfully learn. In this section, we propose a layered Q-learning algorithm, which we derive by exploiting the structure of the transition probability and reward functions in order to decompose the global action-value function into local action-value functions at each layer.Then, in subsection IV.D, we compare the computation, communication, and memory overheads incurred by the centralized and the proposed layered learning algorithms.In this subsection, we first show how the action-value function can be decomposed by exploiting the structure of the considered cross-layer problem. Subsequently, we show how this decomposition leads to a layered Q-learning algorithm, which solves the cross-layer optimization online, in a decentralized manner, when the transition probability and reward functions are unknown a priori. In this section, for notational simplicity, we drop the time slot index n , and denote the next-state 1 n + s using ′ s .The proposed decomposition assumes that (i) each layer has access to the global state s in each time slot; (ii) the APP layer knows the number of states at the joint OS/HW layer, i.e. OS S ; and, (iii) the joint OS/HW layer knows the number of states and actions at the APP layer, i.e. APP S and APP A .Given the additive reward function defined in (15) and the factored transition probability function 15/35 defined in (13), we can rewrite the optimal action-value function defined in (3) as follows: Given the global stateand the optimal state-value function V * , the APP layer can perform the inner computation:which is independent of the immediate action at the joint OS/HW layer (i.e. 1 1 1 1Q * can be computed by repeating this procedure for all ∈ S s .Intuitively, , , Q a s * ′ s from the APP layer, the joint OS/HW layer is able to "fill in" this missing information. Clearly, from the derivation,, , Q a a * s at the joint OS/HW layer is equivalent to the centralized action-value function ( , ) Q * s a .For more information about this decomposition, we refer the interested reader to our prior work [9] , in which we use a similar decomposition to solve the cross-layer optimization offline, in a decentralized manner, under the assumption that the transition probability and reward functions are known a priori.Recall that the centralized Q-learning algorithm described in Section IV.B violates the layered system architecture because it requires a centralized manager to select actions for both of the layers. In contrast, the layered Q-learning algorithm that we propose in this subsection -made possible by the action-value function decomposition described above -adheres to the layered architecture by enabling each layer to autonomously select its own actions and to update its own action-value function. In the following, we discuss the local information requirements for each layer, how each layer selects its local actions, and how each layer updates its local action-value function.The proposed layered Q-learning algorithm requires that the APP layer maintains an estimate of the action-value function on the left-hand side of (20): i.e.,and that the joint OS/HW layer maintains an estimate of the action-value function on the left-hand side of (21): i.e., a ∈ A as described in Section IV.B, but with the greedy action selected as follows:and, the joint OS/HW layer selects its ε -greedy action 1 1 a ∈ A , but with the greedy action selected as follows: , , Q a a s . The message exchanges during the local learning update procedures (described immediately below) serve to "teach" each layer how they impact and are impacted by the other layer.Thus, through the learning process, the layers improve their assumptions about each other, which enables them to improve their greedy actions.After executing the ε -greedy action ( )the system obtains the reward 1 1 2 2 2 2 1 2 1 2 1 , , , n n n n n n n n n n g J V s s Q a s δ ω γ 1 1 2 1 2 2 2 1 1 1 , , , , n n n n n n n n n n Q a s Q a s α δThen, given the scalar ( ) , , , , n n n n n n n n n n J Q a s Q a a δ ωNote that, by using the layered Q-learning algorithm, the layers do not need to directly share their local rewards (i.e. the local learning updates only require the APP layer to know its local reward In this subsection, we compare the computation, communication, and memory overheads associated with the centralized and layered Q-learning algorithms. Fig. 2 and Fig. 3 illustrate the exact information exchanges required for each algorithm; Table II summarizes the greedy action selection procedure and update steps used in each algorithm; and, Table III lists the per time slot computation, communication, and memory overheads associated with each algorithm.Interestingly, the layered Q-learning algorithm is more complex and requires more memory than the centralized algorithm. Note that, in our setting, 1 1 = S A ; hence, the layered Q-learning algorithm incurs approximately twice the computational and memory overheads as the centralized algorithm. The increased overheads can be interpreted as the cost of optimal decentralized learning (optimal in the sense that it performs as well as the centralized learning algorithm). Nevertheless, the proposed centralized and layered Q-learning algorithms are far less complex and use less memory than the alternative learning solutions discussed in Section IV.A. We also observe that, in both cases, the communication overheads per time slot are (1) O because they are independent of the size of the state and action sets at each layer;however, the precise number of messages exchanged depends on the deployed learning algorithm as illustrated in Fig. 2 and Fig. 3 , and noted in Table III. 19/35 2 1 1 1 1 2 , , , , n n n n n n n n n n J Q a s Q a a ω δ, , , , n n n n n n n n n n Q a s Q a s α δ The large number of buffer states at the APP layer significantly limits the system's learning speed because the Q-learning update step defined in (17) updates the action-value function for only one stateaction pair in each time slot. Several existing variants of Q-learning adapt the action-value function for multiple state-action pairs in each time slot. These include model-free temporal-difference-λ updates, and model-based algorithms such as Dyna and prioritized sweeping [14] [15]; however, these existing solutions are not system specific and assume no a priori knowledge of the problem's structure. Consequently, they can only update previously visited state-action pairs, leaving the learning algorithm to act blindly (or with very little information) the first few times that it visits each state, thereby resulting in suboptimal learning performance (our experimental results in Section VI.D strongly support this conclusion).In this subsection, we propose a new Q-learning variant, which is complementary to the 20/35 abovementioned variants. Unlike conventional reinforcement learning algorithms, which assume that no a priori information is available about the problem's dynamics 6 , the proposed algorithm exploits the form of the transition probability and reward functions (defined in Section III) in order to update the action-value function for multiple statistically equivalent 7 state-action pairs in each time slot, including those that have never been visited. In our specific setting, we exploit the fact that the data unit arrival distribution ( | , , )T p t f z h (defined in (9)) is conditionally independent of the current buffer state T p t f z h defined in (9) must be the same for the virtual ETs as it is for the actual ET. In other words, the virtual operating frequency f , the virtual type z , and the virtual configuration h must be the same as the actual operating frequency n f , the actual type n z , and the actual configuration n h , respectively. This also implies that the virtual costs at the APP and joint OS/HW layers are the same as the actual costs at these 6 In conventional reinforcement learning [14] [15], the actors (i.e. the system layers in our setting) are assumed to have no a priori information about the form of the transition probability and reward functions beyond possible high-level structural knowledge about the factored transition and reward dynamics [17] [23]. In other words, in a conventional reinforcement learning framework, the additive decomposition structure of the reward function defined in (1) and the factored transition probability structure defined in (11) may be known a priori, but the actual form of the utility gain defined in (14) and the actual form of the transition probability function defined in (10) cannot be known a priori. 7 We say that a state-action pair ( ) Foresighted decisions are required to optimize the performance of resource-constrained multimedia systems. However, in practical settings, in which the system's dynamics are unknown a priori, efficiently learning the optimal foresighted decision policy can be a challenge. Specifically, selecting an improper learning model can lead to an unacceptably slow learning speed, incur excessive memory overheads, and/or adversely impact the application's real-time performance due to large computational overheads. In this paper, we choose reinforcement learning to appropriately balance time-complexity, memory complexity, and computational complexity. We propose a centralized and a layered reinforcement learning algorithm and extend these algorithms to exploit our partial knowledge of the system's dynamics. The proposed accelerated learning algorithm, which is based on updating multiple statistically equivalent state-action pairs in each time slot, allows us to judiciously tradeoff per-stage computational complexity and time-complexity (i.e. learning speed), while incurring low memory overheads.. In our experimental results, we verify that the layered learning solution achieves the same performance as the centralized solution, which may be impractical to implement if the layers are designed by different manufacturers. We also illustrate that our proposed foresighted learning algorithms outperform the myopic learning algorithm deployed in an existing state-of-the-art cross-layer optimization framework and that, by exploiting knowledge about the system's dynamics, we can significantly outperform an existing application-independent reinforcement learning algorithm that has comparable computational overheads.. In the long term, we believe this work can catalyze a shift in the design and implementation of DMSs (and systems in general) by enabling system layers (modules or components) to proactively reason and interact based on forecasts, as well as evolve, and become smarter, by learning from their interactions.

O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Beyond directed evolution: Darwinian selection as a tool for synthetic biology Abstract Synthetic biology is an engineering approach that seeks to design and construct new biological parts, devices and systems, as well as to re-design existing components. However, rationally designed synthetic circuits may not work as expected due to the context-dependence of biological parts. Darwinian selection, the main mechanism through which evolution works, is a major force in creating biodiversity and may be a powerful tool for synthetic biology. This article reviews selection-based techniques and proposes strict Darwinian selection as an alternative approach for the identification and characterization of parts. Additionally, a strategy for fine-tuning of relatively complex circuits by coupling them to a master standard circuit is discussed.Natural selection is the blind watchmaker, blind because it does not see ahead, does not plan consequences, has no purpose in view. Yet the living results of natural selection overwhelmingly impress us with the appearance of design as if by a master watchmaker, impress us with the illusion of design and planning Richard Dawkins, The blind watchmaker Under changing conditions of life, there is no logical impossibility in the acquirement of any conceivable degree of perfection through natural selection Charles Darwin, On the origin of the speciesSynthetic Biology (SB) is more an approach than a discipline; a framework that includes bio-engineering, systems biology, metabolic engineering and many other disciplines, encompassing the design and construction of new biological parts, devices and systems, as well as the re-design of existing components. SB has also been defined for its potential to embrace the emerging field of designing, synthesizing and evolving new genomes or biomimetic systems. The fundamental principle behind SB is that, analogous to artificial objects, any biological system can be considered as a combination of individual functional elements (de Lorenzo and Danchin 2008) . SB approaches are based on three main principles: part-by-part construction of functional elements or biological parts, standardization of these particular parts, and abstraction of the complex underlying information (e.g. the particular DNA sequence).SB has a very short history; less than a decade in fact. However, the novelty of the discipline is in contrast with the relative antiquity of the tools it uses. In fact, the idea of living organisms as cellular robots might be new, but the techniques used to select, re-design and combine biological parts, in addition to automated sequencing, are standard biotechnological protocols. Unfortunately, experience demonstrates that often rational robot-like, part-by-part approaches simply do not work. For example, Chan et al. (2005) reported that bacteriophages re-designed in order to

O B I B I O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Application of the multi-level time-harmonic fast multipole BEM to 3-D visco-elastodynamics . Application of the multi-level time-harmonic fast multipole BEM to 3-D visco-elastodynamics. Engineering Analysis with Boundary Elements, Elsevier, 2012Elsevier, , 36, pp.744-758. 10.1016Elsevier, /j.enganabound.2011 Abstract This article extends previous work by the authors on the single-and multi-domain time-harmonic elastodynamic multi-level fast multipole BEM formulations to the case of weakly dissipative viscoelastic media. The underlying boundary integral equation and fast multipole formulations are formally identical to that of elastodynamics, except that the wavenumbers are complex-valued due to attenuation. Attention is focused on evaluating the multipole decomposition of the viscoelastodynamic fundamental solution. A damping-dependent modification of the selection rule for the multipole truncation parameter, required by the presence of complex wavenumbers, is proposed. It is empirically adjusted so as to maintain a constant accuracy over the damping range of interest in the approximation of the fundamental solution, and validated on numerical tests focusing on the evaluation of the latter. The proposed modification is then assessed on 3D single-region and multi-region viscoelastodynamic examples for which exact solutions are known. Finally, the multi-region formulation is applied to the problem of a wave propagating in a semi-infinite medium with a lossy semi-spherical inclusion (seismic wave in alluvial basin). These examples involve problem sizes of up to about 3 10 5 boundary unknowns. Elastic wave propagation in unbounded media is a useful idealization for a variety of real-life engineering problems, e.g. railway-or machinery-induced vibrations, seismic wave propagation, or soil-structure interaction. Various domain-based numerical methods, such as the finite difference method [1, 2] , the finite element method [3, 4] and the spectral element method [5, 6] allow to solve elastic wave propagation in three-dimensional unbounded media. They however require the discretization of large 3D regions, leading after discretization to large systems of equations that easily involve millions of degrees of freedom and are prone to cumulative numerical dispersion. Moreover, to avoid the spurious reflected waves induced by artificially truncated domain boundaries, specific techniques such as absorbing boundary conditions [7, 8] , the perfectly matched layers (PMLs) [9] [10] [11] or infinite elements [12] must be implemented. On the other hand, the Boundary Element method (BEM) [13, 14] , in addition to the reduced model size allowed by the boundary-only discretization, is very well suited to dealing with propagation in unbounded media because of the built-in, exact satisfaction of radiation conditions. However, solving the potentially large fully-populated non-symmetric linear system arising in the classical BEM formulation entails unreasonably high CPU time and memory requirements. The fast multipole algorithm is often applied to accelerate iterative solution algorithms applied to the global linear system of BEM equations [15] . The capabilities of the Fast Multipole-Boundary Element Method (FMBEM) and its recursive variant the Multi Level-FMBEM (ML-FMBEM) have rapidly progressed during the last decade, allowing many engineering applications requiring large BEM models, in connection with e.g. acoustics [16] , elastodynamics [17] , Stokes flows [18] or electromagnetism [19] . Fast BEMs have also been proposed using other approaches, in particular based on hierarchical matrices [20] [21] [22] .. FMBEMs for real values of the wavenumber k, which correspond to wave propagation problems in undamped media, have been extensively studied and implemented, see e.g. [23] [24] [25] for background, while the collocation FMBEM for 3D-frequency domain elastodynamics in homogeneous or piecewisehomogeneous elastic media is addressed in [17, 26] . Importantly, these algorithms are backed by error analysis results for the multipole expansion of the fundamental solutions, which also provide guidelines about suitable settings of e.g. truncation parameters.. In contrast, only scattered efforts have so far been devoted to FMMs for Helmholtz-type problems involving complex wavenumbers of the form k ⋆ = k(1 + iβ), with β > 0 quantifying material damping. The overall FMBEM algorithm is identical to that for the corresponding lossless medium, but the setting of important algorithmic parameters such as the truncation parameter is affected by β, an issue which has not been studied in depth, in particular in terms of mathematical error analysis. A complex-wavenumber FMBEM formulation for electromagnetic scattering is proposed in [27, 28] , where the truncation parameter is empirically shown to increase with β in order to maintain a given accuracy on the fundamental solution evaluation. Empirical studies on the low-frequency expansion [29] yield guidelines on the adjustment of the truncation parameter depending on β. The FMBEM analysis of sound fields in porous materials is addressed in [30] , where contributions from cells located beyond a certain normalized distance from the evaluation point are simply ignored, resulting in a well-conditioned influence matrix and a reduced computational time. The latter approach is, however, not suitable for weakly dissipative media (β ≪ 1) because the cut-off distance is proportional to β −1 . This work is concerned with carrying over the single-and multi-domain elastodynamic FMBEM formulations of [17, 26] to the case of weakly dissipative viscoelastic media. Indeed, the ability to consider material attenuation enables more realistic wave propagation modelling, allowing to e.g. reproduce the amplitude decay of propagating waves or free vibrations or describe the finite amplitudes and phase shifts of systems excited near resonance [31] . This article is organized as follows. Section 2 gathers necessary background about the time-harmonic behavior of weakly dissipative viscoelastic media, the relevant BIE formulation and its solution using the ML-FMBEM. Section 3 then concentrates on the crucial task of evaluating the multipole decomposition of the viscoelastodynamic fundamental solution, where a damping-dependent modification of the selection rule for the multipole truncation parameter, required by the presence of complex wavenumbers, is proposed, empirically adjusted so as to maintain a constant accuracy over the damping range of interest in the approximation of the fundamental solution, and validated on numerical tests focusing on the evaluation of the latter. The proposed modification is next assessed in Sec. 4 on full visco-elastodynamic examples for which an exact solution is known. In Sec. 5, the visco-elastodynamic settings are finally applied to the multi-domain version of the ML-FMBEM featuring a BE-BE coupling strategy, previously developed in [17] for the elastodynamic case, and validated on a simple spherically-symmetric configuration and on the problem of a wave propagating in a semi-infinite medium with a lossy semi-spherical inclusion (seismic wave in alluvial basin). In this paper, the application of a previously-published ML-FMBEM for 3-D time-harmonic elastodynamics (involving real wavenumbers) to linear visco-elastodynamics (involving complex wavenumbers, with a small imaginary part accounting for weakly dissipative materials) has been investigated. While the underlying BIE and FM formulations are formally identical to that of elastodynamics, the presence of complex wavenumbers called for a reassessment of the main settings of the ML-FMBEM, and especially of the level-dependent rule for choosing the truncation parameter L. An empirical study has been conducted, showing that a modified, damping-dependent selection rule for L improves the average accuracy for the evaluation of the visco-elastodynamic fundamental solution over a range of material damping ratio values β consistent with intended applications in civil engineering or geophysics. The resulting visco-elastodynamic ML-FMBEM has been tested on several 3D example problems involving either single-domain or multi-domain configurations, with comparisons to known analytical solutions made in both cases. These examples involved BE models of size up to N ≈ 3 10 5 boundary unknowns. Finally, the multi-region formulation was applied to the problem of a (seismic) wave propagating in a semi-infinite medium with a lossy semi-spherical inclusion (seismic wave in alluvial basin). The influence of the truncation radius on the free surface was tested on this problem, using BE models of size up to N ≈ 2 10 5 . As expected, accounting for material damping was found to permit lower values of the truncation spatial radius than in the purely elastic case, implying substantial savings on the BE model size.

O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O