# Over-the-Air Testing of Cognitive Radio Nodes in a Virtual Electromagnetic Environment

Authors: Rajesh K. Sharma; Wim Kotterman; Markus H. Landmann; Christopher Schirmer; Christian Schneider; Frank Wollenschläger; Giovanni Del Galdo; Matthias A. Hein; Reiner S. Thomä
Journal: International Journal of Antennas and Propagation
 (2013)

---

## Abstract

This paper provides an overview of ongoing research
in over-the-air (OTA) testing for next-generation communication
and data transmission devices with special consideration of cognitive radio (CR). Existing state-of-the-art techniques and their merits and limitations are discussed. We identify the requirements and issues for the OTA test in a virtual electromagnetic environment (OTAinVEE) which needs to address the
more complex scenario of future networks, where interference emulation becomes a highly challenging task. A complex interference scenario arises due to the attempt to simultaneously utilize several opportunity dimensions such as frequency, time, direction (space), and polarization, in a dynamic manner in
a multiuser scenario. Although MIMO-OTA testing addresses many limitations of single antenna-conducted test systems, more dimensions and parameters to be addressed in the new scenarios imply further increase in cost and complexity. Closed-loop OTA test setups for CR evaluation are discussed along with an overview of other test scenarios.


---

## Body

## 1. Introduction

There is a constant quest that communication devices in daily use comply with user quality of service (QoS) requirements and regulations. To ensure this, their performance must be tested under realistic operating conditions. Although testing in a real environment, that is, through field tests, provides the full evaluation of the device for that particular environment, such testing should be repeated for each new environment, each new test condition, and for each new device under test (DuT). This is not practical since such tests are very expensive in terms of cost, planning, and effort, reproducibility of the results is difficult to achieve, and experimenting in live radio networks has obvious disadvantages in terms of interference and licensing. OTA testing in the context of this paper is an evaluation method performed in an emulated virtual electromagnetic environment (VEE) created in a shielded chamber, which approximately mimics the properties of a real environment. We, therefore, prefer to term the test system “OTAinVEE.” Throughout this paper, if only the term OTA is used, it should be interpreted as OTAinVEE. Since the test is performed under controlled conditions, arbitrary environments can be emulated in a reproducible way without the risk of interfering operational systems and without the need for licensing.Future-generation radio communication technologies like LTE-A, WiMAX evolution (IEEE802.11m), and WLAN with MIMO (IEEE802.11n) employ MIMO transfer principles for increased throughput. The potential gain of MIMO, by multiple independent data streams at the same frequency and location, can only be realized through the use of multiple antennas on the transmitter and receiver sides of the radio link in combination with the spatiotemporal characteristics of the radio environment. Shunting the antennas like that done in conducted testing removes this vital aspect of MIMO communications. In contrast, in the VEE of OTA test setups, the MIMO capabilities of multiantenna devices can be assessed for realistic environments. Additionally, OTA test systems can be employed for evaluating terminals in networks that have not yet been rolled out.The main use cases for OTA tests considered in this paper are those radio systems that use MIMO transfer, for example, 802.11n or WIMAX mobile, LTE-A and beyond, and cognitive radio (CR) devices. Note that CR functionality for LTE Advanced is already  foreseen, like carrier aggregation. Cognitive devices are listed while, since the second decade of CR research has already begun [1], it is time to pay due attention to OTA evaluation of CR terminals. In this regard, the verification that secondary transmissions of CR devices do not cause detrimental interference to primary users (PU) will be a focus of this paper. Such proof is urgently needed for convincing PU and/or regulators that CR devices will comply with regulations under practical circumstances.In recent years, joint research efforts in MIMO-OTA have been undertaken for standardization purposes for LTE and LTE-A. For example, the Wireless Association (CTIA), the Third-Generation Partnership Project (3GPP RAN WG4), and the European COST2100 and COST IC1004 projects are involved in such activities. The earlier standardized single-input-single-output OTA (SISO-OTA) testing for GSM and UMTS (resulting from similar joint efforts) was relatively straightforward while purely a function of the device, but MIMO-OTA testing has to incorporate the interaction between the propagation characteristics of the radio channel and the receive (Rx) antennas of the DuT [2] and assesses system performance, not only system-relevant device properties. Additional challenges for the test system are posed when addressing the combined effect of multiple antennas and special algorithms that dynamically change the RF performance of the device [3].Important issues in the literature connected to the above-mentioned standardization are channel emulation aspects and simplified test methodologies [4–13]. Also, state-of-the-art techniques were developed and commercial products became available for OTA tests of SISO and MIMO. We will expand on this, in a later section.In contrast, consideration of the uplink (mobile terminal to base station) in OTA testing is generally lacking in the literature. Emulation of the uplink is necessary for assessing the performance of any bidirectional radio communication system, especially when operating in multiuser interference scenarios. In this context, to the best of our knowledge, only few papers on OTA testing of CR equipment have appeared in the literature, and no mature study is available. Emulation of the uplink under realistic channel conditions is crucial for testing CR performance as this is the link on which the DuT may potentially interfere with PU while communicating with other secondary users (SU). This requires accurate emulation of the uplink channels for PU and SU. Of the available papers, an OTA test strategy for evaluating CR nodes has been presented in [14], in which the activities of PU and SU are emulated with respect to received power spectral densities and spatial structures of their radio channels. In order to cope with the complexity, a multilevel test procedure has been proposed to implement different levels of increasing complexity in terms of the operation modes of the DuT, the accuracy of emulation of the radio environment, and the allowed complexity of the test setup. In [15], a test-bed based on a software-defined radio and an OTA test environment has been proposed to analyze the performance of an energy-based sensing algorithm in multiple bands and different directions, enabling reconstruction of PU and SU activities in an authentic radio environment. Realistic OTA simulation of multiuser interference has, however, not been addressed in existing work yet. In this respect, also state-of-the-art equipment has limited resources regarding the number of dimensional variables for spanning emulation space. For instance, most of the setups aim at single-band scenarios with nondirectional resources. Our requirement, however, is a setup for scenarios where multiple communication nodes with multiple transmission policies and standards (PU and SU), equipped with multiple antennas looking for opportunities in multiple directions, coexist at the same geographical location [16, 17]. In sum, the aspects that account for complexity in OTA testing of CR comprise the following:  (i)
simultaneous use of multiple frequency bands, either as intrinsic system feature, for example, in cellular systems or secondary use by cognitive systems,(ii)
multiple users, not necessarily at the same location, meaning that determining the system response for a single user does not suffice as loading the system generally changes its characteristics,(iii)
radio channel characteristics including time-variant multipath propagation for multiple communication entities; in particular systems that do not operate in free space suffer or take benefit from the radio channel variations due to reflection, scattering, diffraction, Doppler shift, and so forth,(iv)
spectral and angular dispersion and spatial/temporal variations experienced by systems that themselves are nonstationary or reside in nonstationary environments,(v)
antenna-environment interactions: as an example, for MIMO systems, the transmit (Tx) and Rx antenna arrays determine to which degree the multiplexing potential of the channel can be realized and/or exploited,(vi)
the activity patterns of different communication entities, for example, spectral occupation, including output power, activity factor and temporal traffic distribution, and spatial distribution. Please note that the consequence of adding all these dimensions to the problem adds to an ever-increasing complexity, which is the main reason to propose the multilevel scheme for OTA testing mentioned earlier [14].The remainder of the paper is organized as follows: Section2 provides a general overview of OTA testing addressing the requirements of the testing solution, followed by Section 3 on state-of-the-art methods and equipment. Section 4 discusses OTA setups for the complex closed-loop test scenario in which both uplink and downlink channels are emulated, which is essential especially for CR evaluation. The hardware and calibration issues are also discussed in this section. Section 5 discusses some other scenarios and use cases for OTA testing and the required modifications. Concluding remarks are given in Section 6.


## 2. OTA Testing Basics

A typical multipath communication scenario with the corresponding parameters of interest is shown in Figure1. To address OTA testing in such a complex scenario, we introduce a layered measurement procedure, schematically drawn in Figure 2. The lowest layer is the OTA installation, including all the mapping of signals onto the OTA channels and antennas. The second layer deals with the propagation layer, differentiating between single-user and multiuser propagations, the latter needing multiple propagation models. In both cases, the propagation models used may differ according to scenario and/or required model complexity. The third layer is the radio layer including the PHY access schemes. It could also distinguish between single-user and multiuser, although the current level of OTA seems still to be single-user. The additional users in the multiuser scenario could be cooperative or interacting ones, but the multiuser influence could also be emulated by additive interference, markedly reducing the emulation overhead. The fourth layer describes the changes in scenario, propagation environment, user activity, user behavior, and so forth. The fifth layer performs overall measurement control, including retrieving performance parameters and producing condensed performance figures. By presenting this layered scheme, some confusion may arise with the well-known standardized OTA procedures for total reference sensitivity (originally: total isotropic sensitivity) and total radiated power in GSM and UMTS. However, these first OTA procedures are not tests of system performance but of (performance relevant) system parameters [18, 19]. We emphasize the system aspect because, as explained in Section 1, OTA is an overall assessment of DuT, like terminals or other radio system entities, not of their constituting parts. One consequence of testing the performance of systems is that the DuT is tested in operational mode and that the tests must be performed in real time. Like mentioned earlier, “operational” also means the tests must be bidirectional, involving both uplink and downlink. Testing radio systems in operational mode calls for emulating radio environments in such a way that, through its antennas, the DuT will not notice the difference between being exposed to emulated fields and operating in a real radio environment. Therefore, as mentioned in Section 1, the OTA emulation should provide a VEE.Figure 1
A typical multipath scenario and parameters to be considered in OTA emulation in a virtual electromagnetic environment (OTAinVEE).Figure 2
A layered measurement procedure for OTA emulation in a virtual electromagnetic environment (OTAinVEE).A typical implementation of an OTA test setup for the single-user downlink scenario for mobile communication user equipment (UE) is depicted in Figure3. Scenario control and signal generation are taken care of by a base-station emulator or mobile communications test set that provides system specific base-band measurement sequences, including the scenario-specific protocol handling, that are modulated onto an appropriate carrier. Then, channel emulators (real-time wideband time-variant filter devices) impose temporal dispersion onto the output signals of test set, much like in conducted tests. In an alternative embodiment, the signal exchange between test set and channel emulators is at base-band level, requiring the channel emulators to provide the generation of RF signals. The channel emulators drive the OTA antennas that, stemming from the similarity with the near-field antenna measurement practice, are often referred to as “probes.” The OTA antennas radiate the fields that through superposition have to render a well-defined resultant field in the center of the antenna configuration. The field emulation will be discussed later. Because of the potentially very wide angular distributions of the multipath components in the mobile channel, OTA antenna arrays typically are annular. In order to allow protocol handling/system control and to transport measurement results of the UE performance back to the measurement device, some form of return channel must be provided. Usually, this is realized by a small pick-up antenna outside the OTA ring, effectively a line-of-sight (LoS) uplink.Figure 3
A typical implementation of an OTAinVEE test setup for the single-user downlink scenario for mobile communication UE.An important parallel exists between virtual electromagnetic environments and acoustical wave field synthesis. The latter creates a virtual acoustic reality, in which the position and character of sound sources can be manipulated at will, as well as the acoustic properties of the room [20]. When done well, a human observer is not able to tell whether the soundscape is artificial or real. One major difference, though, is that acoustic perception is a subjective one, whereas in electromagnetics, objective evaluation is required, which demands objective metrics for emulation accuracy. On the other hand, this does not mean that in a VEE, the emulation accuracy metric is independent of the properties of the DuT. In particular the effective aperture of the antennas of the DuT, determining its angular resolution, should be taken into consideration. For instance, fast variations over angle in the emulated field, that is, field components with high spatial frequencies, will not excite strong responses from small apertures (e.g., of the order of a wavelength or smaller). As a result, emulation errors at high spatial frequencies are less important than errors at the lower spatial frequencies.


### 2.1. Emulation of Wave Fields in Downlink Scenarios

When considering OTA emulation of complex incoming wave fields for downlink scenarios, two main emulation principles can be discerned:(1)
Wave field synthesis (WFS), emulating accurate wave fronts,(2)
emulation of (second-order) spatiotemporal statistics of the wave field.


#### 2.1.1. Wave Field Synthesis

With the first principle, by the use of wave field synthesis, the field is emulated in an anechoic chamber as a superposition of discrete wave components; such decomposition is well known for the modeling of mobile radio channels. Wave fronts with a particular direction are emulated by coherent superposition of elementary waves, radiated by several antennas, each with different delays and complex amplitudes, as illustrated by Figure4. The units controlling the signal fed to the antennas are channel emulators/fading generators similar to those used in conducted tests, one for every OTA antenna.Figure 4
Wave field synthesis for OTA emulation in a virtual electromagnetic environment (OTAinVEE).


#### 2.1.2. Generation of Statistical Moments of the Field

The second emulation principle builds on a popular practice in fading generators for (software) simulation of mobile communication systems and is applied to OTA testing of mobile equipment too. There are two implementations of this approach. One is the “prefaded signals synthesis” in setups very similar to those for wave field synthesis. A proprietary form of it described by Kyösti et al. aims at reproducing the spatial correlation function for the specific scenario [8], by generating random components, with an optimization procedure to make sure the delay and angular spread of the scenario are achieved with special attention for approximating the spatial correlation function.The second implementation is by using reverberation chambers (Figure5), preferably of the stirred-mode type in which the internal standing wave patterns or mode configurations, are continuously and stochastically changed [21]. Averaged over time, the angular spectrum of incoming waves is isotropic. With the aid of additional channel emulators, the power delay profile can be shaped, but how to shape the angular distributions is still an open question [2, 22]. Proponents say averaging over measurement time resembles the DuT response to an averaged field with effectively isotropic incidence (hence, a Rayleigh distribution), but, actually, the result is taken to be the average over many DuT responses to different instantaneous angular spectra, which is not necessarily the same, as the DuT response is strongly non linear. A clear advantage is the lower cost, as the chamber is considerably smaller and fewer generators are required. Emulating the spatiotemporal statistics of radio fields is well suited to deal with highly complex, stochastic fields, but as the statistics are controlled instead of the fields, control is lost over the actual angular spectrum which makes it not suited for testing directional sensing.Figure 5
Reverberation chambers with multiple cavities. Reproduced by permission of EMITE Ing.


#### 2.1.3. Other Methods

Apart from the above-mentioned division, a hybrid principle is proposed, also called the two-stage method [4]. It is applicable in cases where the antenna characteristics of a DuT can be determined independently. This is by no means trivial, especially for small devices; see [4] for details. Determination of the antenna characteristics is the first stage. Then, by projecting the time-variant spatiotemporal distributions of the radio environment onto the antenna pattern(s), the time-variant filter is derived that is used to convolve the transmitted data with in order to render the antenna port signals for this particular radio environment, including DuT orientation and data stream(s). The antenna port signals are fed, in a conducted way, to the antenna ports of the DuT, bypassing the antennas. This comprises the second stage. This method is especially suited for larger DuT that do not fit the OTA installation, but only in case antenna patterns can be determined properly. One clear advantage over full OTA is that fewer resources are needed; for instance, no special chambers are required.Additionally, for mobile communications a two-channel method is proposed [5], but the philosophy behind it is that DuT performance results from the combined performance of antenna(s), front-end, and base-band signal processing. As the latter two purportedly are extremely well tested and probably are very similar among competing products, the aim is assessing antenna performance only, not system performance. In short, according to this philosophy, the second stage of the two-stage method is superfluous. Anyhow, application of the two-channel method to, for instance, CR scenarios is not obvious.


### 2.2. Testing MIMO Systems in Complex Scenarios

The major difference between OTA testing of MIMO systems and SISO systems is the increased complexity. For each transmitter antenna of a MIMO system, the appropriate incoming fields on the DuT need to be emulated. Because these fields are different, as are the data streams, the required amount of resources for WFS or prefaded signal synthesis has to be multiplied with the number of transmitter antennas. Earlier it was remarked that multiuser (MU) interference could be modeled as an additive stochastic process, but in scenarios for cooperative MIMO, at least additional BS fields have to be emulated. This raises the overall complexity again. MU scenarios in which multiuser interference cancellation techniques are to be tested call for realistic emulation of other users, require similar emulation strategies. Modeling the interference as a spatially colored stochastic process might not be realistic enough, but even the spatial coloring will demand additional emulation resources. Yet another increase is to be expected with multiband operation, especially if different OTA antennas are needed for the different bands. For emulation in anechoic chambers, the transition from SISO to MIMO will not require additional OTA antennas as long as the sweet spot (i.e., the area over which the field quality can be guaranteed) is commensurate with the size of the DuT. In case also the uplink has to be emulated, yet another increase in complexity is required, as will be discussed later.For reverberation chambers, more OTA antennas would definitively be necessary. On the other hand, it is still open to research whether MIMO devices can be properly tested in reverberation chambers, one of the main reasons being the difficulty to shape the angular distributions and then multiple of them, according to the properties of transmit antennas and environment [2, 22].


### 2.3. Channel Models

When speaking about channel models in OTA testing, mainly the same models appear that were also used in the development phase for simulations and often also are the models specified for conformance testing. In the present standardization of MIMO-OTA, the SCME model [23] is a point in case. The model was initially designed by the WINNER project group as an interim solution to aid development of beyond-3G systems until the release of the WINNER model. One of its features is the 100 MHz bandwidth instead of the 5 MHz of the preceding SCM model [24], another that the frequency range runs from 2 to 5 GHz, and a third that the two model scenarios, for urban macrocellular and urban microcellular environments, both have one 6-cluster constellation prescribed; essentially meaning that only one single drop can be generated. The model is two dimensional and single polarized. OTA implementations for OTA ring arrays of eight antennas exist, but the possibilities of using only six OTA antennas have already been investigated too [25]. Not the number of OTA antennas is the problem, but the number of channel emulators equaling the number of OTA antennas. Apart from the anechoic room, the channel emulators are the most expensive parts. Note that emulation of a 2 × 2 system in a dual-polarized setup with 6 antennas still requires 24 channel emulators (6 antennas times 2 BS fields times 2 polarization directions). Because of this multiplication by two when going from single polarized to dual polarized, the MIMO-OTA community is reluctant to adopt dual-polarized emulation, although the comprehensive channel models WINNER [26] or COST2100 [27] both are full polarimetric. On the other hand, the huge complexity of these models, in terms of the number of individual delay components that have to be emulated, is the reason they are not used in OTA testing. The same applies to measurements, although another reason is that good measurement data are scarce [28].The comprehensive models also account for the 3D spatial structure of the channel, one that further increases the emulation overhead. In this case, the increase in overhead is caused by the spatial domain, as many more OTA antennas are needed for covering (a part of) a sphere, each requiring a separate channel emulator. However, some attempts have been made to use small 3D-like OTA arrays for increasing the elevation spread of incoming fields, emulating modified IMT-advanced channel models [29]. Predictably, the achievable sweet spot was rather small [30]. On the other hand, a recent study based on measurements shows that simplifications from 3D dual-polarized channels to 2D dual polarized channels, under circumstances, do not greatly impair the realism of the emulation. The two effects in the emulated channel are loss of rank and loss of power. The loss of rank does not seem to be a problem for small MIMO systems, but the loss of power can only be compensated under conditions of perfect power control [31]. However, especially when reducing the 3D full-polarimetric measured data to 2D single polarized, a significant difference in performance can be observed.


## 2.1. Emulation of Wave Fields in Downlink Scenarios

When considering OTA emulation of complex incoming wave fields for downlink scenarios, two main emulation principles can be discerned:(1)
Wave field synthesis (WFS), emulating accurate wave fronts,(2)
emulation of (second-order) spatiotemporal statistics of the wave field.


### 2.1.1. Wave Field Synthesis

With the first principle, by the use of wave field synthesis, the field is emulated in an anechoic chamber as a superposition of discrete wave components; such decomposition is well known for the modeling of mobile radio channels. Wave fronts with a particular direction are emulated by coherent superposition of elementary waves, radiated by several antennas, each with different delays and complex amplitudes, as illustrated by Figure4. The units controlling the signal fed to the antennas are channel emulators/fading generators similar to those used in conducted tests, one for every OTA antenna.Figure 4
Wave field synthesis for OTA emulation in a virtual electromagnetic environment (OTAinVEE).


### 2.1.2. Generation of Statistical Moments of the Field

The second emulation principle builds on a popular practice in fading generators for (software) simulation of mobile communication systems and is applied to OTA testing of mobile equipment too. There are two implementations of this approach. One is the “prefaded signals synthesis” in setups very similar to those for wave field synthesis. A proprietary form of it described by Kyösti et al. aims at reproducing the spatial correlation function for the specific scenario [8], by generating random components, with an optimization procedure to make sure the delay and angular spread of the scenario are achieved with special attention for approximating the spatial correlation function.The second implementation is by using reverberation chambers (Figure5), preferably of the stirred-mode type in which the internal standing wave patterns or mode configurations, are continuously and stochastically changed [21]. Averaged over time, the angular spectrum of incoming waves is isotropic. With the aid of additional channel emulators, the power delay profile can be shaped, but how to shape the angular distributions is still an open question [2, 22]. Proponents say averaging over measurement time resembles the DuT response to an averaged field with effectively isotropic incidence (hence, a Rayleigh distribution), but, actually, the result is taken to be the average over many DuT responses to different instantaneous angular spectra, which is not necessarily the same, as the DuT response is strongly non linear. A clear advantage is the lower cost, as the chamber is considerably smaller and fewer generators are required. Emulating the spatiotemporal statistics of radio fields is well suited to deal with highly complex, stochastic fields, but as the statistics are controlled instead of the fields, control is lost over the actual angular spectrum which makes it not suited for testing directional sensing.Figure 5
Reverberation chambers with multiple cavities. Reproduced by permission of EMITE Ing.


### 2.1.3. Other Methods

Apart from the above-mentioned division, a hybrid principle is proposed, also called the two-stage method [4]. It is applicable in cases where the antenna characteristics of a DuT can be determined independently. This is by no means trivial, especially for small devices; see [4] for details. Determination of the antenna characteristics is the first stage. Then, by projecting the time-variant spatiotemporal distributions of the radio environment onto the antenna pattern(s), the time-variant filter is derived that is used to convolve the transmitted data with in order to render the antenna port signals for this particular radio environment, including DuT orientation and data stream(s). The antenna port signals are fed, in a conducted way, to the antenna ports of the DuT, bypassing the antennas. This comprises the second stage. This method is especially suited for larger DuT that do not fit the OTA installation, but only in case antenna patterns can be determined properly. One clear advantage over full OTA is that fewer resources are needed; for instance, no special chambers are required.Additionally, for mobile communications a two-channel method is proposed [5], but the philosophy behind it is that DuT performance results from the combined performance of antenna(s), front-end, and base-band signal processing. As the latter two purportedly are extremely well tested and probably are very similar among competing products, the aim is assessing antenna performance only, not system performance. In short, according to this philosophy, the second stage of the two-stage method is superfluous. Anyhow, application of the two-channel method to, for instance, CR scenarios is not obvious.


## 2.1.1. Wave Field Synthesis

With the first principle, by the use of wave field synthesis, the field is emulated in an anechoic chamber as a superposition of discrete wave components; such decomposition is well known for the modeling of mobile radio channels. Wave fronts with a particular direction are emulated by coherent superposition of elementary waves, radiated by several antennas, each with different delays and complex amplitudes, as illustrated by Figure4. The units controlling the signal fed to the antennas are channel emulators/fading generators similar to those used in conducted tests, one for every OTA antenna.Figure 4
Wave field synthesis for OTA emulation in a virtual electromagnetic environment (OTAinVEE).


## 2.1.2. Generation of Statistical Moments of the Field

The second emulation principle builds on a popular practice in fading generators for (software) simulation of mobile communication systems and is applied to OTA testing of mobile equipment too. There are two implementations of this approach. One is the “prefaded signals synthesis” in setups very similar to those for wave field synthesis. A proprietary form of it described by Kyösti et al. aims at reproducing the spatial correlation function for the specific scenario [8], by generating random components, with an optimization procedure to make sure the delay and angular spread of the scenario are achieved with special attention for approximating the spatial correlation function.The second implementation is by using reverberation chambers (Figure5), preferably of the stirred-mode type in which the internal standing wave patterns or mode configurations, are continuously and stochastically changed [21]. Averaged over time, the angular spectrum of incoming waves is isotropic. With the aid of additional channel emulators, the power delay profile can be shaped, but how to shape the angular distributions is still an open question [2, 22]. Proponents say averaging over measurement time resembles the DuT response to an averaged field with effectively isotropic incidence (hence, a Rayleigh distribution), but, actually, the result is taken to be the average over many DuT responses to different instantaneous angular spectra, which is not necessarily the same, as the DuT response is strongly non linear. A clear advantage is the lower cost, as the chamber is considerably smaller and fewer generators are required. Emulating the spatiotemporal statistics of radio fields is well suited to deal with highly complex, stochastic fields, but as the statistics are controlled instead of the fields, control is lost over the actual angular spectrum which makes it not suited for testing directional sensing.Figure 5
Reverberation chambers with multiple cavities. Reproduced by permission of EMITE Ing.


## 2.1.3. Other Methods

Apart from the above-mentioned division, a hybrid principle is proposed, also called the two-stage method [4]. It is applicable in cases where the antenna characteristics of a DuT can be determined independently. This is by no means trivial, especially for small devices; see [4] for details. Determination of the antenna characteristics is the first stage. Then, by projecting the time-variant spatiotemporal distributions of the radio environment onto the antenna pattern(s), the time-variant filter is derived that is used to convolve the transmitted data with in order to render the antenna port signals for this particular radio environment, including DuT orientation and data stream(s). The antenna port signals are fed, in a conducted way, to the antenna ports of the DuT, bypassing the antennas. This comprises the second stage. This method is especially suited for larger DuT that do not fit the OTA installation, but only in case antenna patterns can be determined properly. One clear advantage over full OTA is that fewer resources are needed; for instance, no special chambers are required.Additionally, for mobile communications a two-channel method is proposed [5], but the philosophy behind it is that DuT performance results from the combined performance of antenna(s), front-end, and base-band signal processing. As the latter two purportedly are extremely well tested and probably are very similar among competing products, the aim is assessing antenna performance only, not system performance. In short, according to this philosophy, the second stage of the two-stage method is superfluous. Anyhow, application of the two-channel method to, for instance, CR scenarios is not obvious.


## 2.2. Testing MIMO Systems in Complex Scenarios

The major difference between OTA testing of MIMO systems and SISO systems is the increased complexity. For each transmitter antenna of a MIMO system, the appropriate incoming fields on the DuT need to be emulated. Because these fields are different, as are the data streams, the required amount of resources for WFS or prefaded signal synthesis has to be multiplied with the number of transmitter antennas. Earlier it was remarked that multiuser (MU) interference could be modeled as an additive stochastic process, but in scenarios for cooperative MIMO, at least additional BS fields have to be emulated. This raises the overall complexity again. MU scenarios in which multiuser interference cancellation techniques are to be tested call for realistic emulation of other users, require similar emulation strategies. Modeling the interference as a spatially colored stochastic process might not be realistic enough, but even the spatial coloring will demand additional emulation resources. Yet another increase is to be expected with multiband operation, especially if different OTA antennas are needed for the different bands. For emulation in anechoic chambers, the transition from SISO to MIMO will not require additional OTA antennas as long as the sweet spot (i.e., the area over which the field quality can be guaranteed) is commensurate with the size of the DuT. In case also the uplink has to be emulated, yet another increase in complexity is required, as will be discussed later.For reverberation chambers, more OTA antennas would definitively be necessary. On the other hand, it is still open to research whether MIMO devices can be properly tested in reverberation chambers, one of the main reasons being the difficulty to shape the angular distributions and then multiple of them, according to the properties of transmit antennas and environment [2, 22].


## 2.3. Channel Models

When speaking about channel models in OTA testing, mainly the same models appear that were also used in the development phase for simulations and often also are the models specified for conformance testing. In the present standardization of MIMO-OTA, the SCME model [23] is a point in case. The model was initially designed by the WINNER project group as an interim solution to aid development of beyond-3G systems until the release of the WINNER model. One of its features is the 100 MHz bandwidth instead of the 5 MHz of the preceding SCM model [24], another that the frequency range runs from 2 to 5 GHz, and a third that the two model scenarios, for urban macrocellular and urban microcellular environments, both have one 6-cluster constellation prescribed; essentially meaning that only one single drop can be generated. The model is two dimensional and single polarized. OTA implementations for OTA ring arrays of eight antennas exist, but the possibilities of using only six OTA antennas have already been investigated too [25]. Not the number of OTA antennas is the problem, but the number of channel emulators equaling the number of OTA antennas. Apart from the anechoic room, the channel emulators are the most expensive parts. Note that emulation of a 2 × 2 system in a dual-polarized setup with 6 antennas still requires 24 channel emulators (6 antennas times 2 BS fields times 2 polarization directions). Because of this multiplication by two when going from single polarized to dual polarized, the MIMO-OTA community is reluctant to adopt dual-polarized emulation, although the comprehensive channel models WINNER [26] or COST2100 [27] both are full polarimetric. On the other hand, the huge complexity of these models, in terms of the number of individual delay components that have to be emulated, is the reason they are not used in OTA testing. The same applies to measurements, although another reason is that good measurement data are scarce [28].The comprehensive models also account for the 3D spatial structure of the channel, one that further increases the emulation overhead. In this case, the increase in overhead is caused by the spatial domain, as many more OTA antennas are needed for covering (a part of) a sphere, each requiring a separate channel emulator. However, some attempts have been made to use small 3D-like OTA arrays for increasing the elevation spread of incoming fields, emulating modified IMT-advanced channel models [29]. Predictably, the achievable sweet spot was rather small [30]. On the other hand, a recent study based on measurements shows that simplifications from 3D dual-polarized channels to 2D dual polarized channels, under circumstances, do not greatly impair the realism of the emulation. The two effects in the emulated channel are loss of rank and loss of power. The loss of rank does not seem to be a problem for small MIMO systems, but the loss of power can only be compensated under conditions of perfect power control [31]. However, especially when reducing the 3D full-polarimetric measured data to 2D single polarized, a significant difference in performance can be observed.


## 3. State-of-the-Art Products and Test Methods in MIMO-OTA

### 3.1. Brief Literature Review

The major test challenge for MIMO-OTA is creating a repeatable scenario which accurately reflects the MIMO antenna radiation performance in a realistic wireless propagation environment. MIMO-OTA methods differ in the way to reproduce a specified MIMO channel model. In [5], two complementary metrics and corresponding measurement procedures for evaluation of MIMO-OTA performance were presented, in order to address the diversity of possible propagation scenarios. Measurement results from preliminary implementations, including comparison between different LTE devices, were provided.The channel modeling for MIMO-OTA for small multiple antenna devices along with realistic test challenge has been discussed in [6]. The devices considered were relatively small devices: typically UE, terminal, or laptop device. For the practical performance measurement 802.11n device with 2 × 2 MIMO was used as a DuT. In [7], the viability of MIMO-OTA test method in terms of creating an appropriate propagation environment in an anechoic chamber was verified in terms of amplitude distribution of fading coefficients, power delay profile, Doppler spectrum, and spatial autocorrelation. The results were compared to theoretical and reference model characteristics considering radio channel models 3GPP SCM, SCME, and IEEE 802.11n or simplifications of them. The authors claimed good agreement between OTA measurements and the reference model in most cases.In [8], the authors indicated that emulation of a propagation environment inside an anechoic chamber requires unconventional radio channel modeling, namely, a specific mapping of the original models onto the probe antennas. Two methods to generate fading emulator channel coefficients were introduced: the prefaded signals synthesis and the plane-wave synthesis, and simulation results were provided to verify both methods. The plane-wave synthesis for MIMO-OTA was described there as an extension to disciplines of acoustics and electromagnetic plane-wave synthesis as considered in [9]. The authors showed that the geometric description is a prerequisite for the original channel model.As an important issue related to the cost and complexity of the test system, the authors in [10] reviewed the required number of probes for synthesizing the desired fields inside the multiprobe system and proposed some rules as a function of the test zone size for certain uncertainty levels of the field synthesis. Similarly, [11] presents a theoretical study of the generation of the plane-wave conditions for the MIMO-OTA test zone with a finite number of antennas which lie in the horizontal plane on a circle around the test zone. Relations between the size of the test zone and the accuracy of the plane-wave field due to a finite number of antennas were presented considering different angles of arrival of the plane wave. The authors showed that in the case where the OTA antennas are located in the 90° azimuth angle region with 45° intervals (3 antennas), the test zone diameter is smaller than if the antennas are located in the 360° azimuth angle region with 45° intervals (8 antennas). In [9], the aliasing effect caused by a small number of antennas available in MIMO-OTA setups was discussed.In wave field synthesis for MIMO-OTA test setups with anechoic chamber, it was reported in [32] that the effective volume of the test zone increases with respect to the shape of the emulated wave fronts through the use of small vertical arrays. Also, the decay of the emulated field over distance has been found reduced. It was suggested to use passive networks for exciting the array elements, such that no additional channel emulators are needed.In [12], emulation methodology of multiple cluster channels for OTA testing was investigated using a reverberation chamber. This methodology has been used along with the implementation of the single-input-multiple-output (SIMO) LTE standard. It consists of evaluating effective diversity gain level of SIMO LTE orthogonal frequency division multiplex (OFDM) system for different channel models according to the received power, by establishing an active link between the transmitter and the receiver. Significant improvements attained by the implemented system compared to SISO were reported.Identifying appropriate figures-of-merit is an important aspect for MIMO-OTA evaluation. In [13], some new figures-of-merit, aimed at serving the purpose of evaluating the operators’ top priorities for MIMO-OTA compliance testing, were provided based on the LTE MIMO-OTA round robin data from 3GPP. The new figures are MIMO throughput effectiveness, MIMO device throughput effectiveness, MIMO throughput gain, and MIMO device throughput gain.Apart from the literature, there exist some commercial products and solutions on MIMO-OTA testing. Here, we discuss a few of them regarding their methods and available features. Some important specifications and features are also summarized in Table1 for convenience. Please note that the specifications in the table are just for information rather than for comparison since a fair comparison is not possible due to their unique characteristics.Table 1
Summary of some available OTA test solutions.
Solution developer
Chamber type and dimension
Hardware support (available antennas/probes)
Channel model
Frequency range
Other features
Elektrobit [33]
Anechoic
2
×
2 to 8×4, up to 32 channels with 48 fading paths per fading channel available
Standard (LTE, ITU 3G), optional including LTE-A evaluation models, dynamic spatially evolving channel modeling
30–2700 MHz, RF interface channel signal BW 40 MHz
Supports carrier aggregation up to 160 MHz, multiple RF bands, coordinated multipoint, and relaying
Spirent [34]
Anechoic
4 to 32 probe layouts
Standard (3GPP, SCM/SCME, WINNER, and ITU) and classical user-defined models
Up to 6 GHz, signal BW 26 MHz (SR5500 Wireless Channel Emulator)
Transmit antenna having single as well as dual polarizations
SATIMO [35]
Anechoic
4 to 32 MIMO dual-polarized probes
Propagation models in a controlled environment (single cluster, multiple cluster, and uniform).
400 MHz to 6 GHz
Positioner feature allowing to choose desired azimuth for testing the DuT in more orientations
Rohde and Schwarz [5, 36]
Anechoic (5m×5m×5m)
2
×
2 MIMO downlink
Static channel model
Test results reported for 10 MHz BW (LTE mode) in 0.7 GHz band (TS8991 MIMO-OTA)
Quad-ridged horn antennas, dedicatedφ and θ antenna connectors
ETS-Lindgren [37]
Anechoic (4.9m×4.9m×3.7m)
Not specified
Specially modified spatial channel models, appropriate Doppler and delay spreads to emulate the scattering effect of fixed and moving objects
700 MHz to 10 GHz (Model AMS-8700)
MIMO dual-polarized environment simulation antennas
Emite Ing [38]
Multimode-stirred reverberation chambers
8
×
8 MIMO Analyzer (Series E100) having ability to test up to 8 radiating element prototypes
Angle of arrival (AoA), angle spread (AS), number of multipath components (MPC), and the eigenvalues considered as channel parameters
Not specified
Different fading scenarios with and without the presence of the user, fast and nonexpensive
Agilent Technologies [39]
Reverberation, anechoic
2
×
2 MIMO
Correlation-based and geometry-based model
Not specified
Two-stage method


### 3.2. Commercial Products

#### 3.2.1. Elektrobit

It is a MIMO-OTA test solution where flexible configuration for SISO, MISO/SIMO, and MIMO up to 4 × 4 or 8 × 4 are available. It supports 3GPP Release 10 and Release 11 enhancements including carrier aggregation up to 160 MHz, multiple RF bands, coordinated multipoint, and relaying. It provides the channel modeling applications for MIMO, beamforming, multiuser MIMO, smart antenna, and virtual drive testing. It provides integrated uplink and downlink signal separation, meaning there is no need for external duplexers. Both unidirectional and bidirectional fading modes are available and the system is compatible within the Electrobit Propsim product family [33].


#### 3.2.2. Spirent

This MIMO-OTA solution determines the downlink MIMO-OTA performance using an anechoic chamber. The software facilitates comprehensive performance characterization including antenna, RF front-end, and base-band signal processing implementations. Through the characterization of antenna gain or efficiency, branch imbalance, and antenna correlation, for dual-polarized antenna conditions, it enables to precisely characterize the difference between a good and a poor design [34].


#### 3.2.3. SATIMO

Measurement capabilities of their solution [35] include evaluation of receiver diversity-based handsets, evaluation of MIMO performances of WiFi, LTE and WiMAX-based handsets, and emulation of widely standardized 3GPP channel. It employs propagation models in a controlled environment, namely, single cluster, multiple cluster, and uniform. Emulation of variable angles of arrival, angular spread, cross polar ratio (XPR), Doppler, and delay spread are supported. The figures-of-merit are the device throughput in controlled fading environments, channel capacity and bit error rate, and antenna-related parameter characterization such as correlation and diversity gain.


#### 3.2.4. Rohde and Schwarz

The two-channel method was implemented by Rohde and Schwarz for verification of the OTA performance on MIMO devices with a focus on downlink (DL) 2 × 2 MIMO testing for spatial multiplexing and transmit diversity [5, 36]. Measurements of receiver sensitivity and throughput were evaluated with statistical metrics.The TS8991 MIMO-OTA test system supporting the two-channel method consists of an OTA chamber having three angular positioners to control angles, two test antennas (downlink), and one circularly polarized communication antenna (uplink) integrated in the azimuth positionar. Furthermore, the access panel permits five RF connections to the test antennas placed in the chamber. Two quad-ridged horn antennas are utilized as test antennas, each of which is capable of creating orthogonal components of linearly polarized field.


#### 3.2.5. ETS-LINDGREN

ETS-LINDGREN developed a MIMO-OTA test system (model AMS-8700) for a multipath environment. The simulated environment is suitable for evaluation of downlink MIMO performance for emerging wireless technologies such as LTE, WiMAX, and 802.11n Wi-Fi as well as receiving diversity performance of existing wireless technologies [37].The system consists of a dual-polarized antenna array in an absorber lined fully anechoic chamber, connected to technology-specific communication test equipment through a spatial channel emulator. The antenna array transmits downlink signals from a range of angles of arrival (AoA). The spatial channel emulator uses specially modified spatial channel models to feed each antenna in the array with a statistical sampling of the source signal(s) with appropriate Doppler and delay spreads, to emulate the scattering effect of fixed and moving objects. A positioning system allows the wireless DuT to be rotated through the generated field structure, to determine its relative performance in different orientations [37].


#### 3.2.6. EMITE Ing

They have provided a solution for MIMO measurements with a second-generation multimode-stirred chamber. A MIMO measurement service with the 8 × 8 MIMO Analyzer Series E100 is available where up to 8 radiating element prototypes can be tested. Diversity gain, MIMO capacity, efficiency, and other parameters are provided for a variety of fading scenarios with and without the presence of the user. The developers claim that testing with this service could be the only fast and nonexpensive alternative for novel LTE or WiMAX prototypes [38].


#### 3.2.7. Agilent Technologies

The company is mainly involved in research and development of MIMO-OTA with focus on channel models and test methods for measuring the performance of MIMO devices in realistic environments. Results were presented for three OTA methods: the reverberation chamber method, anechoic chamber OTA, and two-stage OTA [39]. The results revealed that although the anechoic chamber method has the flexibility of emulating a variety of multipath conditions and does not require access to the user equipment antenna ports, it can become highly complex and costly to implement especially when dual-polarized measurements are required. This method also has the longest calibration and measurement times as well as the largest chamber requirements compared to other systems. It has been suggested to use a smaller number of probe antennas for reduced complexity and cost, but this makes system accuracy sensitive to calibration and measurement errors [39].


### 3.3. Noncommercial Test Facilities

There exist also some noncommercial facilities aimed mainly at research in OTA testing. Here we discuss the facilities available in Ilmenau. The important specifications and features of these facilities are also summarized in Table2 for convenience.Table 2
Summary of the OTA test facilities in Ilmenau.
Test facility
Chamber type and dimension
Hardware support (available antennas/probes)
Channel emulation
Frequency range
Other features
FORTE [40]
Anechoic (5m×4.5m×3.1m)
12
×
32 (physical channels)
Time domain processing similar to tap delay line (32 taps per physical channel) and frequency domain processing (unlimited fading paths per channel).
350 MHz to 3 GHz (for4×4 channels up to 18 GHz), signal BW 80 MHz
Noncommercial, diverse applications including satellite and terrestrial communications
VISTA [41]
Anechoic (16m×12m×9m), max. DuT size=6m
Under research
Dynamically evolving geometry-based stochastic channel model
70 MHz to 6 GHz
Noncommercial, combining EMC measurements with OTA testing and vehicular channel emulation


#### 3.3.1. Facility for Over-the-Air Research and Testing (FORTE)

Within a collaboration between Ilmenau University of Technology and the Fraunhofer Institute for Integrated Circuits, a Facility for Over-the-Air Research and Testing (FORTE) has been built in Ilmenau which hosts two research platforms in satellite and terrestrial communication, called “SatCom” and “MIMO-OTA” test beds. The MIMO-OTA research is aimed mainly at OTA test methodologies and also meant to actively engage in standardization of OTA practices. Other parts of the research are into wave field synthesis and evaluation of CR nodes. A clear application is to study real-time antenna characteristics of mobile terminals in an OTA test setting. With the projected equipment, a state-of-the-art OTA installation will become available in near future [40]. Some important specifications for the signal generators for the OTA test laboratory are as follows: the connectivity of the OTA test system is 12 × 32 (physical channels). The signal bandwidth supported is 80 MHz across a frequency range from 350 MHz to 3 GHz (for 4 × 4 channels up to 18 GHz). For CR two frequency bands with larger separation can be used for emulation, but with instantaneous bandwidth of 80 MHz for each band. the RF output power is +10 dBm. For the generation of the delay characteristics two modes are available: (i)
mode 1: time domain processing similar to tap delay line,(ii)
mode 2: frequency domain processing. Mode 1 supports 32 taps per physical channel. The delay accuracy better than 30 ps, phase accuracy better than 0.1°, and channel update rate up to 100 kHz are available.


#### 3.3.2. Virtual Road Simulation and Test Facility at Ilmenau University of Technology (VISTA)

VISTA is a test facility in Ilmenau similar but less comprehensive compared to the OTA measurement facility FORTE described before. One major activity on this facility is the conception and implementation of the “virtual road.” We will discuss more on the motivation and development of this facility in Section5.


## 3.1. Brief Literature Review

The major test challenge for MIMO-OTA is creating a repeatable scenario which accurately reflects the MIMO antenna radiation performance in a realistic wireless propagation environment. MIMO-OTA methods differ in the way to reproduce a specified MIMO channel model. In [5], two complementary metrics and corresponding measurement procedures for evaluation of MIMO-OTA performance were presented, in order to address the diversity of possible propagation scenarios. Measurement results from preliminary implementations, including comparison between different LTE devices, were provided.The channel modeling for MIMO-OTA for small multiple antenna devices along with realistic test challenge has been discussed in [6]. The devices considered were relatively small devices: typically UE, terminal, or laptop device. For the practical performance measurement 802.11n device with 2 × 2 MIMO was used as a DuT. In [7], the viability of MIMO-OTA test method in terms of creating an appropriate propagation environment in an anechoic chamber was verified in terms of amplitude distribution of fading coefficients, power delay profile, Doppler spectrum, and spatial autocorrelation. The results were compared to theoretical and reference model characteristics considering radio channel models 3GPP SCM, SCME, and IEEE 802.11n or simplifications of them. The authors claimed good agreement between OTA measurements and the reference model in most cases.In [8], the authors indicated that emulation of a propagation environment inside an anechoic chamber requires unconventional radio channel modeling, namely, a specific mapping of the original models onto the probe antennas. Two methods to generate fading emulator channel coefficients were introduced: the prefaded signals synthesis and the plane-wave synthesis, and simulation results were provided to verify both methods. The plane-wave synthesis for MIMO-OTA was described there as an extension to disciplines of acoustics and electromagnetic plane-wave synthesis as considered in [9]. The authors showed that the geometric description is a prerequisite for the original channel model.As an important issue related to the cost and complexity of the test system, the authors in [10] reviewed the required number of probes for synthesizing the desired fields inside the multiprobe system and proposed some rules as a function of the test zone size for certain uncertainty levels of the field synthesis. Similarly, [11] presents a theoretical study of the generation of the plane-wave conditions for the MIMO-OTA test zone with a finite number of antennas which lie in the horizontal plane on a circle around the test zone. Relations between the size of the test zone and the accuracy of the plane-wave field due to a finite number of antennas were presented considering different angles of arrival of the plane wave. The authors showed that in the case where the OTA antennas are located in the 90° azimuth angle region with 45° intervals (3 antennas), the test zone diameter is smaller than if the antennas are located in the 360° azimuth angle region with 45° intervals (8 antennas). In [9], the aliasing effect caused by a small number of antennas available in MIMO-OTA setups was discussed.In wave field synthesis for MIMO-OTA test setups with anechoic chamber, it was reported in [32] that the effective volume of the test zone increases with respect to the shape of the emulated wave fronts through the use of small vertical arrays. Also, the decay of the emulated field over distance has been found reduced. It was suggested to use passive networks for exciting the array elements, such that no additional channel emulators are needed.In [12], emulation methodology of multiple cluster channels for OTA testing was investigated using a reverberation chamber. This methodology has been used along with the implementation of the single-input-multiple-output (SIMO) LTE standard. It consists of evaluating effective diversity gain level of SIMO LTE orthogonal frequency division multiplex (OFDM) system for different channel models according to the received power, by establishing an active link between the transmitter and the receiver. Significant improvements attained by the implemented system compared to SISO were reported.Identifying appropriate figures-of-merit is an important aspect for MIMO-OTA evaluation. In [13], some new figures-of-merit, aimed at serving the purpose of evaluating the operators’ top priorities for MIMO-OTA compliance testing, were provided based on the LTE MIMO-OTA round robin data from 3GPP. The new figures are MIMO throughput effectiveness, MIMO device throughput effectiveness, MIMO throughput gain, and MIMO device throughput gain.Apart from the literature, there exist some commercial products and solutions on MIMO-OTA testing. Here, we discuss a few of them regarding their methods and available features. Some important specifications and features are also summarized in Table1 for convenience. Please note that the specifications in the table are just for information rather than for comparison since a fair comparison is not possible due to their unique characteristics.Table 1
Summary of some available OTA test solutions.
Solution developer
Chamber type and dimension
Hardware support (available antennas/probes)
Channel model
Frequency range
Other features
Elektrobit [33]
Anechoic
2
×
2 to 8×4, up to 32 channels with 48 fading paths per fading channel available
Standard (LTE, ITU 3G), optional including LTE-A evaluation models, dynamic spatially evolving channel modeling
30–2700 MHz, RF interface channel signal BW 40 MHz
Supports carrier aggregation up to 160 MHz, multiple RF bands, coordinated multipoint, and relaying
Spirent [34]
Anechoic
4 to 32 probe layouts
Standard (3GPP, SCM/SCME, WINNER, and ITU) and classical user-defined models
Up to 6 GHz, signal BW 26 MHz (SR5500 Wireless Channel Emulator)
Transmit antenna having single as well as dual polarizations
SATIMO [35]
Anechoic
4 to 32 MIMO dual-polarized probes
Propagation models in a controlled environment (single cluster, multiple cluster, and uniform).
400 MHz to 6 GHz
Positioner feature allowing to choose desired azimuth for testing the DuT in more orientations
Rohde and Schwarz [5, 36]
Anechoic (5m×5m×5m)
2
×
2 MIMO downlink
Static channel model
Test results reported for 10 MHz BW (LTE mode) in 0.7 GHz band (TS8991 MIMO-OTA)
Quad-ridged horn antennas, dedicatedφ and θ antenna connectors
ETS-Lindgren [37]
Anechoic (4.9m×4.9m×3.7m)
Not specified
Specially modified spatial channel models, appropriate Doppler and delay spreads to emulate the scattering effect of fixed and moving objects
700 MHz to 10 GHz (Model AMS-8700)
MIMO dual-polarized environment simulation antennas
Emite Ing [38]
Multimode-stirred reverberation chambers
8
×
8 MIMO Analyzer (Series E100) having ability to test up to 8 radiating element prototypes
Angle of arrival (AoA), angle spread (AS), number of multipath components (MPC), and the eigenvalues considered as channel parameters
Not specified
Different fading scenarios with and without the presence of the user, fast and nonexpensive
Agilent Technologies [39]
Reverberation, anechoic
2
×
2 MIMO
Correlation-based and geometry-based model
Not specified
Two-stage method


## 3.2. Commercial Products

### 3.2.1. Elektrobit

It is a MIMO-OTA test solution where flexible configuration for SISO, MISO/SIMO, and MIMO up to 4 × 4 or 8 × 4 are available. It supports 3GPP Release 10 and Release 11 enhancements including carrier aggregation up to 160 MHz, multiple RF bands, coordinated multipoint, and relaying. It provides the channel modeling applications for MIMO, beamforming, multiuser MIMO, smart antenna, and virtual drive testing. It provides integrated uplink and downlink signal separation, meaning there is no need for external duplexers. Both unidirectional and bidirectional fading modes are available and the system is compatible within the Electrobit Propsim product family [33].


### 3.2.2. Spirent

This MIMO-OTA solution determines the downlink MIMO-OTA performance using an anechoic chamber. The software facilitates comprehensive performance characterization including antenna, RF front-end, and base-band signal processing implementations. Through the characterization of antenna gain or efficiency, branch imbalance, and antenna correlation, for dual-polarized antenna conditions, it enables to precisely characterize the difference between a good and a poor design [34].


### 3.2.3. SATIMO

Measurement capabilities of their solution [35] include evaluation of receiver diversity-based handsets, evaluation of MIMO performances of WiFi, LTE and WiMAX-based handsets, and emulation of widely standardized 3GPP channel. It employs propagation models in a controlled environment, namely, single cluster, multiple cluster, and uniform. Emulation of variable angles of arrival, angular spread, cross polar ratio (XPR), Doppler, and delay spread are supported. The figures-of-merit are the device throughput in controlled fading environments, channel capacity and bit error rate, and antenna-related parameter characterization such as correlation and diversity gain.


### 3.2.4. Rohde and Schwarz

The two-channel method was implemented by Rohde and Schwarz for verification of the OTA performance on MIMO devices with a focus on downlink (DL) 2 × 2 MIMO testing for spatial multiplexing and transmit diversity [5, 36]. Measurements of receiver sensitivity and throughput were evaluated with statistical metrics.The TS8991 MIMO-OTA test system supporting the two-channel method consists of an OTA chamber having three angular positioners to control angles, two test antennas (downlink), and one circularly polarized communication antenna (uplink) integrated in the azimuth positionar. Furthermore, the access panel permits five RF connections to the test antennas placed in the chamber. Two quad-ridged horn antennas are utilized as test antennas, each of which is capable of creating orthogonal components of linearly polarized field.


### 3.2.5. ETS-LINDGREN

ETS-LINDGREN developed a MIMO-OTA test system (model AMS-8700) for a multipath environment. The simulated environment is suitable for evaluation of downlink MIMO performance for emerging wireless technologies such as LTE, WiMAX, and 802.11n Wi-Fi as well as receiving diversity performance of existing wireless technologies [37].The system consists of a dual-polarized antenna array in an absorber lined fully anechoic chamber, connected to technology-specific communication test equipment through a spatial channel emulator. The antenna array transmits downlink signals from a range of angles of arrival (AoA). The spatial channel emulator uses specially modified spatial channel models to feed each antenna in the array with a statistical sampling of the source signal(s) with appropriate Doppler and delay spreads, to emulate the scattering effect of fixed and moving objects. A positioning system allows the wireless DuT to be rotated through the generated field structure, to determine its relative performance in different orientations [37].


### 3.2.6. EMITE Ing

They have provided a solution for MIMO measurements with a second-generation multimode-stirred chamber. A MIMO measurement service with the 8 × 8 MIMO Analyzer Series E100 is available where up to 8 radiating element prototypes can be tested. Diversity gain, MIMO capacity, efficiency, and other parameters are provided for a variety of fading scenarios with and without the presence of the user. The developers claim that testing with this service could be the only fast and nonexpensive alternative for novel LTE or WiMAX prototypes [38].


### 3.2.7. Agilent Technologies

The company is mainly involved in research and development of MIMO-OTA with focus on channel models and test methods for measuring the performance of MIMO devices in realistic environments. Results were presented for three OTA methods: the reverberation chamber method, anechoic chamber OTA, and two-stage OTA [39]. The results revealed that although the anechoic chamber method has the flexibility of emulating a variety of multipath conditions and does not require access to the user equipment antenna ports, it can become highly complex and costly to implement especially when dual-polarized measurements are required. This method also has the longest calibration and measurement times as well as the largest chamber requirements compared to other systems. It has been suggested to use a smaller number of probe antennas for reduced complexity and cost, but this makes system accuracy sensitive to calibration and measurement errors [39].


## 3.2.1. Elektrobit

It is a MIMO-OTA test solution where flexible configuration for SISO, MISO/SIMO, and MIMO up to 4 × 4 or 8 × 4 are available. It supports 3GPP Release 10 and Release 11 enhancements including carrier aggregation up to 160 MHz, multiple RF bands, coordinated multipoint, and relaying. It provides the channel modeling applications for MIMO, beamforming, multiuser MIMO, smart antenna, and virtual drive testing. It provides integrated uplink and downlink signal separation, meaning there is no need for external duplexers. Both unidirectional and bidirectional fading modes are available and the system is compatible within the Electrobit Propsim product family [33].


## 3.2.2. Spirent

This MIMO-OTA solution determines the downlink MIMO-OTA performance using an anechoic chamber. The software facilitates comprehensive performance characterization including antenna, RF front-end, and base-band signal processing implementations. Through the characterization of antenna gain or efficiency, branch imbalance, and antenna correlation, for dual-polarized antenna conditions, it enables to precisely characterize the difference between a good and a poor design [34].


## 3.2.3. SATIMO

Measurement capabilities of their solution [35] include evaluation of receiver diversity-based handsets, evaluation of MIMO performances of WiFi, LTE and WiMAX-based handsets, and emulation of widely standardized 3GPP channel. It employs propagation models in a controlled environment, namely, single cluster, multiple cluster, and uniform. Emulation of variable angles of arrival, angular spread, cross polar ratio (XPR), Doppler, and delay spread are supported. The figures-of-merit are the device throughput in controlled fading environments, channel capacity and bit error rate, and antenna-related parameter characterization such as correlation and diversity gain.


## 3.2.4. Rohde and Schwarz

The two-channel method was implemented by Rohde and Schwarz for verification of the OTA performance on MIMO devices with a focus on downlink (DL) 2 × 2 MIMO testing for spatial multiplexing and transmit diversity [5, 36]. Measurements of receiver sensitivity and throughput were evaluated with statistical metrics.The TS8991 MIMO-OTA test system supporting the two-channel method consists of an OTA chamber having three angular positioners to control angles, two test antennas (downlink), and one circularly polarized communication antenna (uplink) integrated in the azimuth positionar. Furthermore, the access panel permits five RF connections to the test antennas placed in the chamber. Two quad-ridged horn antennas are utilized as test antennas, each of which is capable of creating orthogonal components of linearly polarized field.


## 3.2.5. ETS-LINDGREN

ETS-LINDGREN developed a MIMO-OTA test system (model AMS-8700) for a multipath environment. The simulated environment is suitable for evaluation of downlink MIMO performance for emerging wireless technologies such as LTE, WiMAX, and 802.11n Wi-Fi as well as receiving diversity performance of existing wireless technologies [37].The system consists of a dual-polarized antenna array in an absorber lined fully anechoic chamber, connected to technology-specific communication test equipment through a spatial channel emulator. The antenna array transmits downlink signals from a range of angles of arrival (AoA). The spatial channel emulator uses specially modified spatial channel models to feed each antenna in the array with a statistical sampling of the source signal(s) with appropriate Doppler and delay spreads, to emulate the scattering effect of fixed and moving objects. A positioning system allows the wireless DuT to be rotated through the generated field structure, to determine its relative performance in different orientations [37].


## 3.2.6. EMITE Ing

They have provided a solution for MIMO measurements with a second-generation multimode-stirred chamber. A MIMO measurement service with the 8 × 8 MIMO Analyzer Series E100 is available where up to 8 radiating element prototypes can be tested. Diversity gain, MIMO capacity, efficiency, and other parameters are provided for a variety of fading scenarios with and without the presence of the user. The developers claim that testing with this service could be the only fast and nonexpensive alternative for novel LTE or WiMAX prototypes [38].


## 3.2.7. Agilent Technologies

The company is mainly involved in research and development of MIMO-OTA with focus on channel models and test methods for measuring the performance of MIMO devices in realistic environments. Results were presented for three OTA methods: the reverberation chamber method, anechoic chamber OTA, and two-stage OTA [39]. The results revealed that although the anechoic chamber method has the flexibility of emulating a variety of multipath conditions and does not require access to the user equipment antenna ports, it can become highly complex and costly to implement especially when dual-polarized measurements are required. This method also has the longest calibration and measurement times as well as the largest chamber requirements compared to other systems. It has been suggested to use a smaller number of probe antennas for reduced complexity and cost, but this makes system accuracy sensitive to calibration and measurement errors [39].


## 3.3. Noncommercial Test Facilities

There exist also some noncommercial facilities aimed mainly at research in OTA testing. Here we discuss the facilities available in Ilmenau. The important specifications and features of these facilities are also summarized in Table2 for convenience.Table 2
Summary of the OTA test facilities in Ilmenau.
Test facility
Chamber type and dimension
Hardware support (available antennas/probes)
Channel emulation
Frequency range
Other features
FORTE [40]
Anechoic (5m×4.5m×3.1m)
12
×
32 (physical channels)
Time domain processing similar to tap delay line (32 taps per physical channel) and frequency domain processing (unlimited fading paths per channel).
350 MHz to 3 GHz (for4×4 channels up to 18 GHz), signal BW 80 MHz
Noncommercial, diverse applications including satellite and terrestrial communications
VISTA [41]
Anechoic (16m×12m×9m), max. DuT size=6m
Under research
Dynamically evolving geometry-based stochastic channel model
70 MHz to 6 GHz
Noncommercial, combining EMC measurements with OTA testing and vehicular channel emulation


### 3.3.1. Facility for Over-the-Air Research and Testing (FORTE)

Within a collaboration between Ilmenau University of Technology and the Fraunhofer Institute for Integrated Circuits, a Facility for Over-the-Air Research and Testing (FORTE) has been built in Ilmenau which hosts two research platforms in satellite and terrestrial communication, called “SatCom” and “MIMO-OTA” test beds. The MIMO-OTA research is aimed mainly at OTA test methodologies and also meant to actively engage in standardization of OTA practices. Other parts of the research are into wave field synthesis and evaluation of CR nodes. A clear application is to study real-time antenna characteristics of mobile terminals in an OTA test setting. With the projected equipment, a state-of-the-art OTA installation will become available in near future [40]. Some important specifications for the signal generators for the OTA test laboratory are as follows: the connectivity of the OTA test system is 12 × 32 (physical channels). The signal bandwidth supported is 80 MHz across a frequency range from 350 MHz to 3 GHz (for 4 × 4 channels up to 18 GHz). For CR two frequency bands with larger separation can be used for emulation, but with instantaneous bandwidth of 80 MHz for each band. the RF output power is +10 dBm. For the generation of the delay characteristics two modes are available: (i)
mode 1: time domain processing similar to tap delay line,(ii)
mode 2: frequency domain processing. Mode 1 supports 32 taps per physical channel. The delay accuracy better than 30 ps, phase accuracy better than 0.1°, and channel update rate up to 100 kHz are available.


### 3.3.2. Virtual Road Simulation and Test Facility at Ilmenau University of Technology (VISTA)

VISTA is a test facility in Ilmenau similar but less comprehensive compared to the OTA measurement facility FORTE described before. One major activity on this facility is the conception and implementation of the “virtual road.” We will discuss more on the motivation and development of this facility in Section5.


## 3.3.1. Facility for Over-the-Air Research and Testing (FORTE)

Within a collaboration between Ilmenau University of Technology and the Fraunhofer Institute for Integrated Circuits, a Facility for Over-the-Air Research and Testing (FORTE) has been built in Ilmenau which hosts two research platforms in satellite and terrestrial communication, called “SatCom” and “MIMO-OTA” test beds. The MIMO-OTA research is aimed mainly at OTA test methodologies and also meant to actively engage in standardization of OTA practices. Other parts of the research are into wave field synthesis and evaluation of CR nodes. A clear application is to study real-time antenna characteristics of mobile terminals in an OTA test setting. With the projected equipment, a state-of-the-art OTA installation will become available in near future [40]. Some important specifications for the signal generators for the OTA test laboratory are as follows: the connectivity of the OTA test system is 12 × 32 (physical channels). The signal bandwidth supported is 80 MHz across a frequency range from 350 MHz to 3 GHz (for 4 × 4 channels up to 18 GHz). For CR two frequency bands with larger separation can be used for emulation, but with instantaneous bandwidth of 80 MHz for each band. the RF output power is +10 dBm. For the generation of the delay characteristics two modes are available: (i)
mode 1: time domain processing similar to tap delay line,(ii)
mode 2: frequency domain processing. Mode 1 supports 32 taps per physical channel. The delay accuracy better than 30 ps, phase accuracy better than 0.1°, and channel update rate up to 100 kHz are available.


## 3.3.2. Virtual Road Simulation and Test Facility at Ilmenau University of Technology (VISTA)

VISTA is a test facility in Ilmenau similar but less comprehensive compared to the OTA measurement facility FORTE described before. One major activity on this facility is the conception and implementation of the “virtual road.” We will discuss more on the motivation and development of this facility in Section5.


## 4. Closed-Loop OTA Testing

Most of the products discussed in the former section are specifically designed for testing mobile communication systems in the downlink. However, these systems are bidirectional, meaning, and operating over an uplink too. From testing perspective, at least the protocol handling must be guaranteed, but almost any kind of return channel will suffice to get the protocol messages through, for instance, the small pick-up antenna in the anechoic chamber mentioned earlier. However, this will generally not be satisfactory for assessing system performance. In cellular mobile radio, every system design has to pay special attention to the uplink; therefore, replacing the time-variant and dispersive mobile channel in the uplink by a simple LoS link generally is not realistic.


### 4.1. Emulation of the Uplink

Emulation of the uplink is based on the same principle as that of the downlink, but effectively in reverse. Receivers are connected to the OTA antennas instead of transmitters; their outputs are filtered by channel emulators and coherently combined onto the Rx connectors of the secondary communication partner or communication tester. A schematic diagram of a closed-loop OTAinVEE test-bed realization, incorporating the uplink, is given in Figure 6. More description for this realization including hardware and calibration issues is provided later in this section. Note that in case of emulating MIMO systems, similar to the downlink with respect to the number of transmitters, the required number of channel emulators is multiplied by the number of Rx units of the secondary communication partner or test set. Because the uplink uses WFS too, the receivers should operate in a phase-coherent manner. However, in our experience, manufacturers do not easily promise that their channel emulators operate phase coherently and we expect that purchasing a large phase-coherent set of receivers will prove to be as difficult.Figure 6
Schematic diagram of a closed-loop OTA setup (OTAinVEE) with full emulation of spatiotemporal channels in uplink (UL) and downlink (DL).


### 4.2. Practical OTA Setups

Although OTA for LTE-based systems is subject of a large standardization effort, closed-loop testing of LTE is practically impossible at the moment, because no measurement equipment is available that can cope with realistic uplink behavior. As the commercially offered base station (BS) emulators were exclusively designed for connected testing, they are not able to equalize channel fading.


#### 4.2.1. Cognitive Radio Evaluation

The cognitive behavior of the DuT, with respect to relevance for OTA testing, will be interpreted in terms of the cognition cycle as defined by Mitola III and Maguire Jr. [1]. (i)
Observe: a CR node performs spectrum sensing to be aware of the surrounding radio environment.(ii)
Orient and plan: the observed data are subjected to intelligent processing.(iii)
Decide: based on the results of the processing, the CR node decides on spectrum opportunities.(iv)
Act: the CR dynamically exploits the opportunities for transmission and avoids interference to PU.(v)
Learn: learning from previous events is a vital capability of CR.Cognitive abilities numbers (ii), (iii), and (v) are not necessarily tested over the air, but gaining information over these internal states of cognitive nodes has consequences for the OTA setup. Based on the cognitive cycle, we see two different categories of possible OTA tests:(i)
spectrum sensing (observe, orient and plan, and decide),(ii)
secondary link performance (observe, orient and plan, decide, and act).Spectrum Sensing.Spectrum sensing is an important component of the cognitive cycle which decides the presence or absence of primary (licensed) users in the spectrum of interest. Some existing spectrum sensing techniques are matched filtering, cyclostationary-based sensing [42, 43], energy detection (ED) [44], and correlation-based sensing [45]. The performance of each detection method in the given environment is to be evaluated for possible acceptance of the method in real CR applications. The performance of spectrum sensing methods is expressed using the receiver operating characteristics (ROC) which describes the relationship between the probability of correct detection (Pd) and the probability of false alarm (Pfa) [46].In addition to the exploitation of conventional resources like frequency and time, some recent works have also considered direction as a potential resource to be exploited by a CR [17]. The evaluation for different sensing methods using omnidirectional as well as directional sensing can be realistically evaluated by OTA testing. For example, in [15], an evaluation of directional sensing using ED-based sensing algorithm in multiple bands has been presented in terms of ROC. The similar evaluation for more realistic multipath and multiuser environment is desired for possible acceptance of the sensing method utilized by CR.In the case of spectrum sensing, only downlink behavior is tested, so no closed-loop test setup is required. It is even fair to say that the emphasis will be mostly on observe because the other two capabilities can also be tested off-line with software simulations. However, in order to monitor the sensing process, some means of information transfer is needed. Unfortunately, any conductive wire or cable attached to the device will influence the antenna characteristics in a rather unpredictable way. The smaller the device relative to the wavelength, the stronger this influence [47]. Therefore, a wireless, or nongalvanic, data connection is preferred. Using transfer over optical fiber is optimal; otherwise, data have to be stored internally in the device and retrieved off-line while preserving the synchronization between emulated signals and sense data, as this is crucial for evaluation.As depicted in Figure7, there might be several PU whose activity behavior has to be described accurately. Statistical models of PU activity have been published (e.g., [48]) that model the activity of radio nodes by binary spectrum occupancy. The use of a realistic statistical user activity model as an emulation basis has the advantage of avoiding playing back large amounts of recorded data when testing the sensing capabilities of the DuT in time-critical applications [49, 50]. The existing models are based on measurements, but, generally, the temporal resolution of the measurements is too low for accurately parameterizing the temporal properties of cellular bands [48, 51, 52]. However, it is possible to perform measurements with sufficient time resolution and to tune the model to a much higher intrinsic accuracy [53].Cognitive radio in two sensing scenarios: LoS (a) and more realistic multipath (b).
(a)
(b)Secondary Link Performance. The transmissions of CR should not cause harmful interference to PU. Such a system would require SU to accurately detect and react to the varying spectrum usage by the PU network. The effect of secondary communication in the real environment is important to be monitored, which is actually the secondary link performance. In [54], a statistical model for aggregate interference of a cognitive network was proposed. This type of statistical behavior of the cognitive network interference can be tested and verified by using OTA testing. For such a verification, the interference emulation is the most important and challenging task. After the verification of such statistical model, it can be utilized by the OTA test system itself as an information inside the layered measurement procedure described in Figure 2 for further evaluation of the CR node performance. Additionally, any interference mitigation techniques can be evaluated by the system. The ultimate goal is to verify the possibility of coexistence of the primary and CR systems.Uplink verification is associated with the next logical capability of a CR-Act. A test scenario consisting of secondary transmission in addition to sensing in typical multipath environment is shown in Figure8. For this, the adaptive transmission should be tested with features as dynamic frequency and power control, adaptive modulation and coding, and beam-steering. This makes it almost a complete system test and closed-loop testing is indispensable because the uplink is involved. Viable tests regard (i)
the proper exploitation of spectrum opportunities in frequency and/or space,(ii)
the SU tracking of PU with dynamic frequency occupation,(iii)
the sustainable secondary throughput under realistic propagation conditions. These tests have different requirements. The first two call for measuring the instantaneous angular power spectrum of the CR under test while acting. This requires plane-wave analyses (near-field far-field transformations) of the wave fields radiated by the DuT, not necessarily in real time, as off-line processing is well possible). This requires that the DuT is encircled by OTA antennas for reception like in case of the downlink. The third testing of secondary throughput also requires reception of signals in the uplink by encircling antennas, but real-time processing is needed to impose realistic (spatiotemporal) channel fading onto the signals passed to the secondary communication partner. For the moment we assume that the communication partner of the DuT is passive or that the return link can be some simple LoS link inside the anechoic chamber. Providing a realistic secondary return channel will require many more emulator channel, because the downlink now consists of two different streams: the signals from the PU and the (secondary) return link. A very interesting test would also be the real-time analysis of performance degradation of operational PU by SU interference. However, we have no clue how to perform these tests within a single OTA setup as the channel from one PU to his communication partner differs from the one emulated for the DuT.Figure 8
A typical multipath environment to be emulated for the evaluation of CR devices in OTAinVEE closed-loop test-bed [14].A schematic diagram of closed-loop setup of OTAinVEE test-bed realization for CR evaluation is shown in Figure 9.Figure 9
OTAinVEE test-bed realization for closed-loop evaluation of CR nodes.


#### 4.2.2. Practical Setups for LTE (Advanced)

Despite the fact that realistic tests of operational radio systems require emulation of uplink and downlink radio channels, the state of the art is that the largest standardization effort, that for LTE and LTE-A, is still trying to get grip on the essence of MIMO-OTA testing and deals with downlink aspects only. Additionally, in order not to complicate matters further, the link adaptation mechanisms of the system are suppressed by prescribing specific modulation-and-coding schemes.


### 4.3. Emulation Hardware

On the market a wide range of channel emulators are available that strongly differ in price, RF accuracy, and available features. Previously they were designed for conducted testing and therefore most of them are limited in output power, which is actually required for OTA. Furthermore, some of the vendors cannot guarantee phase coherence and stability of their outputs especially at elevated output power. For the emulation of the temporal structure of the channel, one typical implementation is a tapped-delay line with a limited number of possible taps and resolution for each signal path. Every individually emulated wave component requires at least one tap position in a number of signal paths. This approach is resource intensive in radio environments with rich multipath. The presence of dense multipath components (DMC) enriches the multipath situation even further [55], but an emulation with the tapped-delay line approach is almost impossible. Therefore, some vendors also offer channel emulators where the convolution of the communication signals with the propagation channel is performed in the frequency domain. This allows for an emulation of a large number of paths.A further aspect when choosing a channel emulator is its connectivity. For the downlink emulationMBS=2⋯4 inputs and MDn=2⋯32 outputs are commonly available. When considering an uplink emulation with the same quality as that of the downlink, MUp=2⋯32 inputs and MBS=2⋯4 outputs are required. The uplink case is therefore most challenging, but mandatory for the test of cognitive devices exploiting directional degrees-of-freedom.Last but not least, the complexity of the channel (2D, 3D, and full polarization) to be emulated will define the overall connectivity needed for the OTA test setup. As remarked earlier, 3D environments for 2 × 2 MIMO configurations may be simplified to 2D, as long as the polarization characteristics are kept [31], but for larger MIMO configurations such a simplification will affect channel rank too much. In those cases, the resulting dual-polarized 3D emulation would require an extension of the number of generator outputs (in the downlink) and receiver inputs (in the uplink) to up to 200 channels for a sweet spot of 30 cm in diameter, at around 2 GHz. An extension to such a large number of phase coherent outputs is only available from few vendors, whereas such a large number of inputs are not available at all, to our knowledge. In consequence, this means that an uplink observation as required for CR tests may use different techniques such as channel sounding (fast switching over the angular dimension).


### 4.4. Calibration of OTA Setups

One crucial point when using the OTA methods, especially the multi-probe method, is the calibration of the setup as this determines the accuracy of the wave field synthesis [56]. As the wave field synthesis requires a coherent superposition of discrete wave components, the phase and amplitude differences of all signal paths MBS×MDn and MUp×MBS as shown in Figure 6 have to be accounted for in calibration. In the context of phase and amplitude calibration, static and dynamic inaccuracies are relevant. Static differences mainly result from antenna feeds, OTA antennas, antenna positions and antennaalignment, and polarization purity. Dynamic phase and amplitude inaccuracies are mainly caused by phase changes while scanning the sweet spot (bending of RF or even optical fiber cable that is connected to the field probe for calibration) and long-term drift effects of each signal path (MBS×MDn and MUp×MBS) with different drift characteristics. The latter can be reduced to some extent only when carefully considering the temperature stability of the entire cabling arrangement. Note that CR applications additionally require the calibration of each frequency band applied.Recently, during the inter-laboratory/intertechnique comparison test effort of the CTIA MIMO-OTA Special Group, a serious problem was experienced with the definitions of vital concepts of signal-to-noise ratio (SNR) and received power. Although seasoned experts were dealing with these concepts, at least three different implementations with different measurement outcomes resulted. This urged closer investigations.


### 4.5. Channel Modeling Requirements

One key issue for the development and operation of OTA test environments is the provision of application and scenario-specific propagation channel data sets. In the past, various channel models [57] were developed within the European projects COST [58], WINNER [26], or standardization bodies [59]. Furthermore, channel sounding data sets for different evaluation scenarios have gained an increased attraction [60–62]. While first proposals for channel models with focus on vehicle-to-vehicle communication applications [63, 64] have been driven, white spaces can be found in the area of channel models with specific focus to provide broadband directional channel data sets to evaluate CR nodes. A consistent channel model framework for CR applications can be derived from extensions of widely accepted models from 3GPP [59] and WINNER [26].The main features of a channel model suitable for CR testing are multiband, multilink, and multi-scenario. Multiscenario is very interesting, because a CR node located in an outside environment will receive and hence sense signals from transmitters located above roof top (urban macrocell) and/or below roof top (urban microcell) and/or from nodes located inside a building (indoor to outdoor scenario). Furthermore, nodes located within these different scenarios usually use different communications standards and frequency bands. Therefore, the minimum extension of any current available channel model has to be multi-band and multi-scenario. Note, the uplink channel is likely to be similar to the downlink channel but certainly is not identical in frequency-division duplex (FDD) systems. Even with time-division duplex (TDD) systems, when the channel coherence time becomes short in comparison to the duplex time interval, reciprocity is not guaranteed any longer. A special case is CR, as the essence of secondary communications is not interfering with primary communications. Therefore, the RF activity of cognitive (secondary) systems in the uplink should be thoroughly scrutinized; the more so if the cognitive device uses directional antennas to diminish potential interference. Typically, the signals of the PU will be transmitted on the downlink, with the return link carrying the secondary communications. As secondary communication partners are not colocated with primary transmitters, secondary channels differ from primary ones, although they are on the same frequency.


## 4.1. Emulation of the Uplink

Emulation of the uplink is based on the same principle as that of the downlink, but effectively in reverse. Receivers are connected to the OTA antennas instead of transmitters; their outputs are filtered by channel emulators and coherently combined onto the Rx connectors of the secondary communication partner or communication tester. A schematic diagram of a closed-loop OTAinVEE test-bed realization, incorporating the uplink, is given in Figure 6. More description for this realization including hardware and calibration issues is provided later in this section. Note that in case of emulating MIMO systems, similar to the downlink with respect to the number of transmitters, the required number of channel emulators is multiplied by the number of Rx units of the secondary communication partner or test set. Because the uplink uses WFS too, the receivers should operate in a phase-coherent manner. However, in our experience, manufacturers do not easily promise that their channel emulators operate phase coherently and we expect that purchasing a large phase-coherent set of receivers will prove to be as difficult.Figure 6
Schematic diagram of a closed-loop OTA setup (OTAinVEE) with full emulation of spatiotemporal channels in uplink (UL) and downlink (DL).


## 4.2. Practical OTA Setups

Although OTA for LTE-based systems is subject of a large standardization effort, closed-loop testing of LTE is practically impossible at the moment, because no measurement equipment is available that can cope with realistic uplink behavior. As the commercially offered base station (BS) emulators were exclusively designed for connected testing, they are not able to equalize channel fading.


### 4.2.1. Cognitive Radio Evaluation

The cognitive behavior of the DuT, with respect to relevance for OTA testing, will be interpreted in terms of the cognition cycle as defined by Mitola III and Maguire Jr. [1]. (i)
Observe: a CR node performs spectrum sensing to be aware of the surrounding radio environment.(ii)
Orient and plan: the observed data are subjected to intelligent processing.(iii)
Decide: based on the results of the processing, the CR node decides on spectrum opportunities.(iv)
Act: the CR dynamically exploits the opportunities for transmission and avoids interference to PU.(v)
Learn: learning from previous events is a vital capability of CR.Cognitive abilities numbers (ii), (iii), and (v) are not necessarily tested over the air, but gaining information over these internal states of cognitive nodes has consequences for the OTA setup. Based on the cognitive cycle, we see two different categories of possible OTA tests:(i)
spectrum sensing (observe, orient and plan, and decide),(ii)
secondary link performance (observe, orient and plan, decide, and act).Spectrum Sensing.Spectrum sensing is an important component of the cognitive cycle which decides the presence or absence of primary (licensed) users in the spectrum of interest. Some existing spectrum sensing techniques are matched filtering, cyclostationary-based sensing [42, 43], energy detection (ED) [44], and correlation-based sensing [45]. The performance of each detection method in the given environment is to be evaluated for possible acceptance of the method in real CR applications. The performance of spectrum sensing methods is expressed using the receiver operating characteristics (ROC) which describes the relationship between the probability of correct detection (Pd) and the probability of false alarm (Pfa) [46].In addition to the exploitation of conventional resources like frequency and time, some recent works have also considered direction as a potential resource to be exploited by a CR [17]. The evaluation for different sensing methods using omnidirectional as well as directional sensing can be realistically evaluated by OTA testing. For example, in [15], an evaluation of directional sensing using ED-based sensing algorithm in multiple bands has been presented in terms of ROC. The similar evaluation for more realistic multipath and multiuser environment is desired for possible acceptance of the sensing method utilized by CR.In the case of spectrum sensing, only downlink behavior is tested, so no closed-loop test setup is required. It is even fair to say that the emphasis will be mostly on observe because the other two capabilities can also be tested off-line with software simulations. However, in order to monitor the sensing process, some means of information transfer is needed. Unfortunately, any conductive wire or cable attached to the device will influence the antenna characteristics in a rather unpredictable way. The smaller the device relative to the wavelength, the stronger this influence [47]. Therefore, a wireless, or nongalvanic, data connection is preferred. Using transfer over optical fiber is optimal; otherwise, data have to be stored internally in the device and retrieved off-line while preserving the synchronization between emulated signals and sense data, as this is crucial for evaluation.As depicted in Figure7, there might be several PU whose activity behavior has to be described accurately. Statistical models of PU activity have been published (e.g., [48]) that model the activity of radio nodes by binary spectrum occupancy. The use of a realistic statistical user activity model as an emulation basis has the advantage of avoiding playing back large amounts of recorded data when testing the sensing capabilities of the DuT in time-critical applications [49, 50]. The existing models are based on measurements, but, generally, the temporal resolution of the measurements is too low for accurately parameterizing the temporal properties of cellular bands [48, 51, 52]. However, it is possible to perform measurements with sufficient time resolution and to tune the model to a much higher intrinsic accuracy [53].Cognitive radio in two sensing scenarios: LoS (a) and more realistic multipath (b).
(a)
(b)Secondary Link Performance. The transmissions of CR should not cause harmful interference to PU. Such a system would require SU to accurately detect and react to the varying spectrum usage by the PU network. The effect of secondary communication in the real environment is important to be monitored, which is actually the secondary link performance. In [54], a statistical model for aggregate interference of a cognitive network was proposed. This type of statistical behavior of the cognitive network interference can be tested and verified by using OTA testing. For such a verification, the interference emulation is the most important and challenging task. After the verification of such statistical model, it can be utilized by the OTA test system itself as an information inside the layered measurement procedure described in Figure 2 for further evaluation of the CR node performance. Additionally, any interference mitigation techniques can be evaluated by the system. The ultimate goal is to verify the possibility of coexistence of the primary and CR systems.Uplink verification is associated with the next logical capability of a CR-Act. A test scenario consisting of secondary transmission in addition to sensing in typical multipath environment is shown in Figure8. For this, the adaptive transmission should be tested with features as dynamic frequency and power control, adaptive modulation and coding, and beam-steering. This makes it almost a complete system test and closed-loop testing is indispensable because the uplink is involved. Viable tests regard (i)
the proper exploitation of spectrum opportunities in frequency and/or space,(ii)
the SU tracking of PU with dynamic frequency occupation,(iii)
the sustainable secondary throughput under realistic propagation conditions. These tests have different requirements. The first two call for measuring the instantaneous angular power spectrum of the CR under test while acting. This requires plane-wave analyses (near-field far-field transformations) of the wave fields radiated by the DuT, not necessarily in real time, as off-line processing is well possible). This requires that the DuT is encircled by OTA antennas for reception like in case of the downlink. The third testing of secondary throughput also requires reception of signals in the uplink by encircling antennas, but real-time processing is needed to impose realistic (spatiotemporal) channel fading onto the signals passed to the secondary communication partner. For the moment we assume that the communication partner of the DuT is passive or that the return link can be some simple LoS link inside the anechoic chamber. Providing a realistic secondary return channel will require many more emulator channel, because the downlink now consists of two different streams: the signals from the PU and the (secondary) return link. A very interesting test would also be the real-time analysis of performance degradation of operational PU by SU interference. However, we have no clue how to perform these tests within a single OTA setup as the channel from one PU to his communication partner differs from the one emulated for the DuT.Figure 8
A typical multipath environment to be emulated for the evaluation of CR devices in OTAinVEE closed-loop test-bed [14].A schematic diagram of closed-loop setup of OTAinVEE test-bed realization for CR evaluation is shown in Figure 9.Figure 9
OTAinVEE test-bed realization for closed-loop evaluation of CR nodes.


### 4.2.2. Practical Setups for LTE (Advanced)

Despite the fact that realistic tests of operational radio systems require emulation of uplink and downlink radio channels, the state of the art is that the largest standardization effort, that for LTE and LTE-A, is still trying to get grip on the essence of MIMO-OTA testing and deals with downlink aspects only. Additionally, in order not to complicate matters further, the link adaptation mechanisms of the system are suppressed by prescribing specific modulation-and-coding schemes.


## 4.2.1. Cognitive Radio Evaluation

The cognitive behavior of the DuT, with respect to relevance for OTA testing, will be interpreted in terms of the cognition cycle as defined by Mitola III and Maguire Jr. [1]. (i)
Observe: a CR node performs spectrum sensing to be aware of the surrounding radio environment.(ii)
Orient and plan: the observed data are subjected to intelligent processing.(iii)
Decide: based on the results of the processing, the CR node decides on spectrum opportunities.(iv)
Act: the CR dynamically exploits the opportunities for transmission and avoids interference to PU.(v)
Learn: learning from previous events is a vital capability of CR.Cognitive abilities numbers (ii), (iii), and (v) are not necessarily tested over the air, but gaining information over these internal states of cognitive nodes has consequences for the OTA setup. Based on the cognitive cycle, we see two different categories of possible OTA tests:(i)
spectrum sensing (observe, orient and plan, and decide),(ii)
secondary link performance (observe, orient and plan, decide, and act).Spectrum Sensing.Spectrum sensing is an important component of the cognitive cycle which decides the presence or absence of primary (licensed) users in the spectrum of interest. Some existing spectrum sensing techniques are matched filtering, cyclostationary-based sensing [42, 43], energy detection (ED) [44], and correlation-based sensing [45]. The performance of each detection method in the given environment is to be evaluated for possible acceptance of the method in real CR applications. The performance of spectrum sensing methods is expressed using the receiver operating characteristics (ROC) which describes the relationship between the probability of correct detection (Pd) and the probability of false alarm (Pfa) [46].In addition to the exploitation of conventional resources like frequency and time, some recent works have also considered direction as a potential resource to be exploited by a CR [17]. The evaluation for different sensing methods using omnidirectional as well as directional sensing can be realistically evaluated by OTA testing. For example, in [15], an evaluation of directional sensing using ED-based sensing algorithm in multiple bands has been presented in terms of ROC. The similar evaluation for more realistic multipath and multiuser environment is desired for possible acceptance of the sensing method utilized by CR.In the case of spectrum sensing, only downlink behavior is tested, so no closed-loop test setup is required. It is even fair to say that the emphasis will be mostly on observe because the other two capabilities can also be tested off-line with software simulations. However, in order to monitor the sensing process, some means of information transfer is needed. Unfortunately, any conductive wire or cable attached to the device will influence the antenna characteristics in a rather unpredictable way. The smaller the device relative to the wavelength, the stronger this influence [47]. Therefore, a wireless, or nongalvanic, data connection is preferred. Using transfer over optical fiber is optimal; otherwise, data have to be stored internally in the device and retrieved off-line while preserving the synchronization between emulated signals and sense data, as this is crucial for evaluation.As depicted in Figure7, there might be several PU whose activity behavior has to be described accurately. Statistical models of PU activity have been published (e.g., [48]) that model the activity of radio nodes by binary spectrum occupancy. The use of a realistic statistical user activity model as an emulation basis has the advantage of avoiding playing back large amounts of recorded data when testing the sensing capabilities of the DuT in time-critical applications [49, 50]. The existing models are based on measurements, but, generally, the temporal resolution of the measurements is too low for accurately parameterizing the temporal properties of cellular bands [48, 51, 52]. However, it is possible to perform measurements with sufficient time resolution and to tune the model to a much higher intrinsic accuracy [53].Cognitive radio in two sensing scenarios: LoS (a) and more realistic multipath (b).
(a)
(b)Secondary Link Performance. The transmissions of CR should not cause harmful interference to PU. Such a system would require SU to accurately detect and react to the varying spectrum usage by the PU network. The effect of secondary communication in the real environment is important to be monitored, which is actually the secondary link performance. In [54], a statistical model for aggregate interference of a cognitive network was proposed. This type of statistical behavior of the cognitive network interference can be tested and verified by using OTA testing. For such a verification, the interference emulation is the most important and challenging task. After the verification of such statistical model, it can be utilized by the OTA test system itself as an information inside the layered measurement procedure described in Figure 2 for further evaluation of the CR node performance. Additionally, any interference mitigation techniques can be evaluated by the system. The ultimate goal is to verify the possibility of coexistence of the primary and CR systems.Uplink verification is associated with the next logical capability of a CR-Act. A test scenario consisting of secondary transmission in addition to sensing in typical multipath environment is shown in Figure8. For this, the adaptive transmission should be tested with features as dynamic frequency and power control, adaptive modulation and coding, and beam-steering. This makes it almost a complete system test and closed-loop testing is indispensable because the uplink is involved. Viable tests regard (i)
the proper exploitation of spectrum opportunities in frequency and/or space,(ii)
the SU tracking of PU with dynamic frequency occupation,(iii)
the sustainable secondary throughput under realistic propagation conditions. These tests have different requirements. The first two call for measuring the instantaneous angular power spectrum of the CR under test while acting. This requires plane-wave analyses (near-field far-field transformations) of the wave fields radiated by the DuT, not necessarily in real time, as off-line processing is well possible). This requires that the DuT is encircled by OTA antennas for reception like in case of the downlink. The third testing of secondary throughput also requires reception of signals in the uplink by encircling antennas, but real-time processing is needed to impose realistic (spatiotemporal) channel fading onto the signals passed to the secondary communication partner. For the moment we assume that the communication partner of the DuT is passive or that the return link can be some simple LoS link inside the anechoic chamber. Providing a realistic secondary return channel will require many more emulator channel, because the downlink now consists of two different streams: the signals from the PU and the (secondary) return link. A very interesting test would also be the real-time analysis of performance degradation of operational PU by SU interference. However, we have no clue how to perform these tests within a single OTA setup as the channel from one PU to his communication partner differs from the one emulated for the DuT.Figure 8
A typical multipath environment to be emulated for the evaluation of CR devices in OTAinVEE closed-loop test-bed [14].A schematic diagram of closed-loop setup of OTAinVEE test-bed realization for CR evaluation is shown in Figure 9.Figure 9
OTAinVEE test-bed realization for closed-loop evaluation of CR nodes.


## 4.2.2. Practical Setups for LTE (Advanced)

Despite the fact that realistic tests of operational radio systems require emulation of uplink and downlink radio channels, the state of the art is that the largest standardization effort, that for LTE and LTE-A, is still trying to get grip on the essence of MIMO-OTA testing and deals with downlink aspects only. Additionally, in order not to complicate matters further, the link adaptation mechanisms of the system are suppressed by prescribing specific modulation-and-coding schemes.


## 4.3. Emulation Hardware

On the market a wide range of channel emulators are available that strongly differ in price, RF accuracy, and available features. Previously they were designed for conducted testing and therefore most of them are limited in output power, which is actually required for OTA. Furthermore, some of the vendors cannot guarantee phase coherence and stability of their outputs especially at elevated output power. For the emulation of the temporal structure of the channel, one typical implementation is a tapped-delay line with a limited number of possible taps and resolution for each signal path. Every individually emulated wave component requires at least one tap position in a number of signal paths. This approach is resource intensive in radio environments with rich multipath. The presence of dense multipath components (DMC) enriches the multipath situation even further [55], but an emulation with the tapped-delay line approach is almost impossible. Therefore, some vendors also offer channel emulators where the convolution of the communication signals with the propagation channel is performed in the frequency domain. This allows for an emulation of a large number of paths.A further aspect when choosing a channel emulator is its connectivity. For the downlink emulationMBS=2⋯4 inputs and MDn=2⋯32 outputs are commonly available. When considering an uplink emulation with the same quality as that of the downlink, MUp=2⋯32 inputs and MBS=2⋯4 outputs are required. The uplink case is therefore most challenging, but mandatory for the test of cognitive devices exploiting directional degrees-of-freedom.Last but not least, the complexity of the channel (2D, 3D, and full polarization) to be emulated will define the overall connectivity needed for the OTA test setup. As remarked earlier, 3D environments for 2 × 2 MIMO configurations may be simplified to 2D, as long as the polarization characteristics are kept [31], but for larger MIMO configurations such a simplification will affect channel rank too much. In those cases, the resulting dual-polarized 3D emulation would require an extension of the number of generator outputs (in the downlink) and receiver inputs (in the uplink) to up to 200 channels for a sweet spot of 30 cm in diameter, at around 2 GHz. An extension to such a large number of phase coherent outputs is only available from few vendors, whereas such a large number of inputs are not available at all, to our knowledge. In consequence, this means that an uplink observation as required for CR tests may use different techniques such as channel sounding (fast switching over the angular dimension).


## 4.4. Calibration of OTA Setups

One crucial point when using the OTA methods, especially the multi-probe method, is the calibration of the setup as this determines the accuracy of the wave field synthesis [56]. As the wave field synthesis requires a coherent superposition of discrete wave components, the phase and amplitude differences of all signal paths MBS×MDn and MUp×MBS as shown in Figure 6 have to be accounted for in calibration. In the context of phase and amplitude calibration, static and dynamic inaccuracies are relevant. Static differences mainly result from antenna feeds, OTA antennas, antenna positions and antennaalignment, and polarization purity. Dynamic phase and amplitude inaccuracies are mainly caused by phase changes while scanning the sweet spot (bending of RF or even optical fiber cable that is connected to the field probe for calibration) and long-term drift effects of each signal path (MBS×MDn and MUp×MBS) with different drift characteristics. The latter can be reduced to some extent only when carefully considering the temperature stability of the entire cabling arrangement. Note that CR applications additionally require the calibration of each frequency band applied.Recently, during the inter-laboratory/intertechnique comparison test effort of the CTIA MIMO-OTA Special Group, a serious problem was experienced with the definitions of vital concepts of signal-to-noise ratio (SNR) and received power. Although seasoned experts were dealing with these concepts, at least three different implementations with different measurement outcomes resulted. This urged closer investigations.


## 4.5. Channel Modeling Requirements

One key issue for the development and operation of OTA test environments is the provision of application and scenario-specific propagation channel data sets. In the past, various channel models [57] were developed within the European projects COST [58], WINNER [26], or standardization bodies [59]. Furthermore, channel sounding data sets for different evaluation scenarios have gained an increased attraction [60–62]. While first proposals for channel models with focus on vehicle-to-vehicle communication applications [63, 64] have been driven, white spaces can be found in the area of channel models with specific focus to provide broadband directional channel data sets to evaluate CR nodes. A consistent channel model framework for CR applications can be derived from extensions of widely accepted models from 3GPP [59] and WINNER [26].The main features of a channel model suitable for CR testing are multiband, multilink, and multi-scenario. Multiscenario is very interesting, because a CR node located in an outside environment will receive and hence sense signals from transmitters located above roof top (urban macrocell) and/or below roof top (urban microcell) and/or from nodes located inside a building (indoor to outdoor scenario). Furthermore, nodes located within these different scenarios usually use different communications standards and frequency bands. Therefore, the minimum extension of any current available channel model has to be multi-band and multi-scenario. Note, the uplink channel is likely to be similar to the downlink channel but certainly is not identical in frequency-division duplex (FDD) systems. Even with time-division duplex (TDD) systems, when the channel coherence time becomes short in comparison to the duplex time interval, reciprocity is not guaranteed any longer. A special case is CR, as the essence of secondary communications is not interfering with primary communications. Therefore, the RF activity of cognitive (secondary) systems in the uplink should be thoroughly scrutinized; the more so if the cognitive device uses directional antennas to diminish potential interference. Typically, the signals of the PU will be transmitted on the downlink, with the return link carrying the secondary communications. As secondary communication partners are not colocated with primary transmitters, secondary channels differ from primary ones, although they are on the same frequency.


## 5. Related Test Scenario: C2C/C2I Communication—Issues and Developments

Recently, channel measurements and test scenarios for Car-to-Car (C2C) and Car-to-Infrastructure (C2I) communications have gained large attraction. The term “Car” can be more generalized by using “vehicl.” The embedding of vehicles as nodes into ad hoc and meshed network structures for safety, security, fleet and traffic management, and communication seems to be one key technology for future. Channel models and test methods have to consider the specific requirements of those network structures. The following features play an important role for any considered channel model approach.(i)
Continuous time evolution of the channel: because of the rapidly varying C2C/C2I channel, a continuous evolvement of the channel has to be introduced. With that the models can cover extended evaluation time series.(ii)
Realistic node traffic model: in order to cover long evaluation time series, realistic vehicle traffic models are necessary.(iii)
Joint clusters for different links and scenarios: to introduce realistic time evolution and correlations between emulated multilinks, joint or common clusters between links and even between scenarios (e.g., urban macro and micro) are required.(iv)
Modeling of realistic interference situations: an open issue is the modeling of realistic interference and coexistence situations. In particular, in rapidly changing vehicular scenarios, various and time-dependent interferences from different networks and/or other noncooperative sources (audio and sensing links within vehicles) have to be expected.Considering high standards for safety and security, the testing of interference management and suppression algorithms for coexistence present major challenges of C2C/C2I communications.C2C/C2I communications undergo a very rapid development, for example, alluding to the ITS-G5 standard. The issues of coexistence and interference resulting from a great number of equivalent communicating nodes in ad hoc and time-variant networks cannot be overestimated. In addition to propagation aspects and channel modeling, conventional and even more so electric cars comprise a multitude of electric and electronic systems for driver assistance, safety systems, data transmission, intravehicular networks, and radio connectivity to a multitude of services extending from tens of MHz well into the GHz range. Related to these rapid developments, issues of coexistence, interference, inter-operability, and quality assurance demand for measurement techniques, not only reliable conditions irrespective of time, place, region, or climate conditions, but also accounting for the specificities of vehicles. For C2C/C2I communications, accurate and reproducible measurement techniques are as important as or even more important than the multipath wave field synthesis. Typical scenarios can include from tens up to 200 cars, which can never be tested on real road, especially not in repeated manner for changing situations.The Thuringian Center of Innovation in Mobility (ThIMo) at Ilmenau University of Technology [41] pursues a holistic approach towards solving such challenges related to mobility, with an explicit focus on information and communication technologies. The VISTA test facility is similar but less comprehensive compared to the FORTE described before. This installation aims at combining traditional antenna and electromagnetic compatibility measurement approaches with OTA testing and vehicular channel emulation. Fitting to and complementing with the research already ongoing at the FORTE in Ilmenau, alternative techniques are to be developed, to create synthetic radio environments and evaluate their interdependence with the vehicle, road infrastructure, and driver. The key objective of the approach is to emulate realistic radio scenarios as realistic as possible, possibly in real-time, under well-defined partial or fully anechoic conditions.The first stage of the VISTA test facility comprises a semianechoic chamber and a chassis dynamometer integrated into a turntable. The dynamometer enables emulation of realistic driving scenarios synchronized with electromagnetic synthesis and measurement hardware in addition to electromagnetic compatibility (EMC) measurements of vehicles and powertrain components under operation. Some of the key parameters of the planned implementation are mentioned here. The chamber size is 16 m × 12 m × 9 m and the maximum DuT size is 6 m. The frequency range of operation is 70–6000 MHz, where the velocity up to 120 Km/h has been simulated.


## 6. Conclusions

This paper has provided an overview of an over-the-air test in virtual electromagnetic environment (OTAinVEE) for future communication devices, with the main focus on CR devices operating in highly complex environments with multiple dimensions of parameters, each having multiple elements. A brief survey of state-of-the-art methods and products has been presented. Closed-loop OTAinVEE has been discussed with issues related to hardware, calibration, and channel modeling. Some practical OTA scenarios including CR testing and setups have also been presented. Although closed-loop OTA testing is costly and complex, it is inevitable for successful implementation of future systems and for guaranteeing their performance.



---

*Source: 945283-2013-08-06.xml*