Design, Analysis, and Learning Control of a
Fully Actuated Micro Wind Turbine
J. Zico Kolter, Zachary Jackowski, Russ Tedrake*
Abstract— Wind power represents one of the most promisingsources of renewable energy, and improvements to wind turbine
design and control can have a significant impact on energy
sustainability. In this paper we make two primary contributions: first, we develop and present a actuated micro windturbine intended for research purposes. While most academic
work on wind turbine control has largely focused on simulated
evaluations, most turbine simulators are quite limited in theirability to model unsteady aerodynamic effects induced by theturbine; thus, there is a huge value to validating wind turbine
control methods on a physical system, and the platform wepresent here makes this possible at a very low cost. The second
contribution of this paper a novel policy search method, applied
to optimizing power production in Region II wind speeds.
Our method is similar in spirit to Reinforcement Learning
approaches such as the REINFORCE algorithm, but explicitly
models second order terms of the cost function and makes
efficient use of past execution data. We evaluate this methodon the physical turbine and show it it is able to quickly andrepeatably achieve near-optimal power production within abouta minute of execution time and without any a priori model ofthe system.

I. INTRODUCTION
Energy issues pose one of the greatest challenges facing
society. More than 86% of the world’s energy currently
comes from (unsustainable) fossil fuels, and worldwide energy demand continues to grow rapidly [1]. Wind power
represents one of the most promising sources of renewableenergy: currently wind is more economically feasible thansolar or biomass for electricity generation, with some projections predicting wind, given good environmental conditions
and proper government subsidies, to be of similar cost
to fossil fuels for electricity generation [2]. Despite thispromise, significant improvements in the deployment andcontrol of wind turbines are needed if wind is to contributea significant portion of electricity worldwide [3]. One particularly important challenge, which we consider in this paper,is extracting maximum power in “Region II” wind speeds,
where the turbine is not operating at its rated power [4];since efficient operations in such speeds can potentially allow
for wind turbines in many more locations than currently
possible, addressing this problem is crucial for continued
expansion of wind power [5].
In this paper, we make two primary contributions. First,
we develop and present a fully actuated variable pitch,
variable speed micro wind turbine (shown in Figure 1)
with power sensing, load control, and individual blade pitch*All authors are with the Computer Science and Artificial Intelligence
Laboratory, Massaschusetts Institute of Technology, Cambridage MA, 02139{kolter,zackbass,russt}@csail.mit.edu.

control, along with a software architecture for controllingthe system. While simulators naturally play a large role inwind research due to the impracticality and cost of many
control experiments on full-scale turbines, there are also
considerable limitations to wind turbine simulators: a turbine
under non-uniform incoming flow or employing periodic
pitch control can induce unsteady aerodynamic flows [6],whereas most common simulators are based on quasi-steady
aerodynamic assumptions. Thus, there is a huge value towind experiments conducted on physical systems, since thesecan validate control methods beyond what is possible withsimulation alone; indeed, as we illustrate in this paper, evensteady-state power predictions from a standard simulation
tool are quite far from what is actually produced by our
turbine. While micro turbines are of course much smallerscale that utility-sized, empirical results on such platforms
are a natural complement to the simulation-based results thatcurrently dominate from academic turbine control work.
The second contribution of this paper is a trust-region
method for online policy optimization, applied on the turbine to maximize power output in Region II winds speeds.
Our algorithmic method builds upon reinforcement learningalgorithms for stochastic policy gradient methods [7] (alsocalled likelihood ratio stochastic gradient methods [8]), but
explicitly includes second-order terms that approximate theHessian of the cost function in order to take larger steps
the policy parameter space. By using past data via importance sampling methods, the algorithm is able to efficiently
build local second order estimates of the cost function, andthen find the parameters that optimize this cost function.
We compare the method to existing reinforcement learningtechniques, and show that the algorithm is able to quickly
and reliably steer the turbine operating parameters to thosethat maximize its power production.
II. BACKGROUND AND RELATED WORKWind energy has received a great deal of attention inrecent years, and there is a vast amount of work on thedesign, modeling, and control of wind turbines. A full review
of relevant literature is beyond the scope of this paper,and we focus instead only on those elements of turbine
design and control that are most relevant to the presentation
here. However, there are several introductory texts on windenergy in general, and we refer the reader to these for more
information [4], [9]. A number of papers survey more recent
control work and challenges [10], [11].
The piece of wind turbine control work most closely
related to this paper is work on power optimization usingsearch methods we consider is their batch formulation: themethods typically consider several executions of a policyon the real system, and then use this experience to update
the parameters of the policy by approximately computing
gradient or other update information. Of the above methods,
this is most similar to iterative learning control, but the focus
of such research has been on more on improving probabilistic
estimates of the gradient (e.g. [25]) and devising different
optimization-based approaches to using these gradient (e.g.
[26]) rather than notion of closed loop stability that are
more common in ILC. Fundamentally speaking, though, thepolicy search methods have the same goal as ILC, and thealgorithms we discuss could certainly be viewed as ILC
methods.
Fig. 1.

Picture of our micro wind turbine in operation.

III. HARDWARE AND SOFTWARE DESIGN

collective pitch control. Most initial work in this area waslargely model based, and focused on methods for accurately
tracking the optimal control setpoints under changing conditions [12], [13], [14]. Alternatively, newer work has focusedon model-free methods for optimizing power coefficients, using pitch control, via extremum seeking control [15] or finite
difference methods [16]. These approaches are conceptuallysimilar to the proposed approach here, but are strictly firstorder methods, do not efficiently reuse past trajectories, andwere tested only in simulation.
Since this paper is about a physical platform, we also
briefly discuss turbine controls work on real hardware.
The vast majority of research in advanced wind control
techniques (including all papers cited above ) is performed
entirely in simulation: the ubiquity of standard simulation
packages such as FAST [17] and WT Perf [18], as well as thecost involved with work on a a full-scale turbine, have madefull system real-world experiments a rarity in wind research.
Notable exceptions to this include work on the Controls
Advanced Research Turbine (CART) [19], including workon load reduction [20] and individual pitch control [21], andwork on adaptive control for Region 2 power optimization(but without pitch control). However, with the exceptionof this last paper, work on the CART has focused largely
on model-based approaches, largely due to the fact thatsignificant effort has been invested in ensuring accurate
models for this particular turbine; furthermore, the CART
is a relatively large-scale system, and not replicable by mostresearch labs for testing purposes. There is also some workon smaller scale turbines [22], but we know of no such
systems with active pitch control.
From a controls perspective, our online learning methodbuilds most directly upon the REINFORCE algorithm [7]and likelihood ratio gradient estimation [8]. As stochastic
methods that seek to improve control performance via interaction with the system, these methods are closely related
to adaptive control techniques in general, and also iterative
learning control (ILC) [23] and extremum seeking control[24]. Informally speaking, a characteristic of the policy

We begin by discussing the physical turbine platform wehave developed for this work. Our platform is a variable
pitch, variable speed wind turbine, with a blade radius of 1.52
meters and a maximum power output of 300W DC. The system is based upon the Extractor turbine designed by Alternate
Power Technologies, Inc.1 , but with significant modifications.
We replaced the mechanical blade pitch mechanism of theExtractor turbine with servo motors attached to each blade
root; this allows us to individually control the turbine blades
at very high frequencies. We transmit data and power signals
to the servos using a slip ring in the turbine nacelle andtrack the rotor angle using an encoder off the main rotor
shaft. Offboard, the power generated by the system is fed
into a board that both monitors the power output by thesystem, and can programmably vary its resistance (this inturn produces more or less torque on the generator). Finally,a simple wind tunnel built using commodity fans powers
the turbine itself. All the elements are controlled by an
offboard computer via the Lightweight Communication andMarshaling (LCM) software [27]. A block diagram of thesecomponents is shown in Figure 2.
One element we want to highlight is the total cost of
the system, which is approximately $3,250 for the turbine
and associated hardware ($1,000 for the Extractor turbine,$1,550 for the Dynamixel servos, $350 for encoder andslip ring, $250 for the power monitoring hardware and$100 for various cables and wiring) plus $2,600 for thewind tunnel ($2,100 for fans, and $500 for lumber). We
are not aware of any low-cost micro turbines that have thecapabilities of our system (individual pitch control, variable
speed operation); thus, we feel the platform itself has thepotential to significantly impact small and academic-scale
research in wind power. We are happy to share detailed
design specifications with anyone wishing to build a similarsystem.
A. Hub and Nacelle Design
The turbine hub and nacelle house the pitch actuators,
rotor encoder, and generator, as shown in Figure 3. To control
1 http://www.vpturbines.com

Fig. 2.

Block diagram of the hardware components in the micro turbine.

Blade
TimingBelt

Main Shaft
Encoder

Hub

Fig. 5. Internals of the turbine nacelle, showing the generator, encoder andslip ring.

Generator
Slip Ring
Pitch Servo

Fig. 3.

DC Power

Mechanical diagram of the turbine hub and nacelle.

blade pitch, we use three Dynamixel EX-106+ servo motors2
attached to the blade roots. The servos are daisy chained
together and controlled digitally via an RS-485 link, whichallows for setpoint control at 250 Hz with less than 1ms
latency (in addition to the onboard PD controller in the
servos). A picture of the hub and servo is shown in Figure
4.
Since the hub itself rotates, to provide data and power tothe servos we use a Mercotac 430 slip ring3 in the nacelle,
and run the wires through the rotor shaft. The nacelle also
houses a US Digital HB5M encoder4 off the main shiftused to track the orientation of the rotor (necessary both2 http://www.robotis.com/xe/dynamixel en
3 http://www.mercotac.com
4 http://www.usdigital.com

Fig. 4. Front view of the turbine hub and a closeup of the servo motor
used for pitch control.

Fig. 6. Circuit diagram (left) and photograph (right) of the programmableresistor and power monitoring board.

to estimate speed and to synchronize blade orientation withrotor angle) as well as the generator itself, a Delco 12SI
alternator modified for use with the Extractor turbine. The
components of the nacelle are shown in Figure 5.
B. Power Control and Monitoring
To monitor and regulate the power output by the turbine
we attach a resistor to the DC power output lines, and monitor current and voltage using a Phidgets USB A/D converter
and associated voltage and current measuring devices.5 In
order to regulate the power output it is important to be able
to vary the effective resistance: lower resistance will impose
a larger torque on the generator, and some intermediate (butunknown) resistance is needed to maximize power output.
Thus, we place two high-power resistors (0.31 and 5 Ohm
respectively) in series, and short the later with a MOSFET
controlled by a PWM signal at 25kHz. By varying the duty
cycle of the PWM from 0 to 1, we can smoothly interpolate
between a resistance of 5.31 and 0.31 Ohms. The circuit
diagram for this device, as well as a photograph of the board,is shown in Figure 6.
5 http://www.phidgets.com

Fig. 7.
turbine.

Block diagram of the software architecture for controlling the

Fig. 8.

Rear view of the wind tunnel used for powering the turbine.

C. Software Architecture
The turbine is controlled via an offboard computer, running a software system built upon the LCM message passing
framework. LCM offers an attractive architecture for building
such a system, as the different drivers and controllers can be
developed in a modular fashion, and the system has builtin logging and playback capabilities. The basic softwarearchitecture is shown in Figure 7. In addition to the threedriver modules that manage the servos, the encoder, and thepower board, we use two basic controllers: 1) a collective
pitch that controls all the blades equally, using a simple
integral control loop (although the servos internally havea PD controller, the friction in the drive train and steady
aerodynamic forces often cause these to have a steady state
error, which we correct via an integral controller), and 2) an
independent pitch controller that synchronizes blade pitchof the different blades with rotor angle using feed-forward
compensation; this element will be discussed further in
Section V-B. Also illustrated in Figure 7 is a policy search
procedure that we will describe in Section V-C, which uses
collective pitch control and feedback from the power board
to optimize energy output, and a MATLAB graphical userinterface that allows for easy visualization and control of thedifferent modules.
D. Wind Tunnel
To power the turbine, we constructed a small circular windtunnel using wood and commodity commercial fans. The
tunnel is eight feet long (a four foot contraction section anda four foot straight section), with an output diameter of 1.72
meters (20 cm larger than the diameter of the turbine), andgenerates an average wind speed of 6.5 m/sec over this area
(approximately 13.5 miles per hour). This is not fast enoughto reach the rated power output of 300W, but is still able togenerate a significant amount of power at high rotational
speeds. A photograph of the fans powering the tunnel isshown in Figure 8.
IV. TRUST REGION POLICY SEARCH
The primary control task we consider in this paper is maximizing power production in wind speeds below the turbine’s
rated power; concretely, by adjusting the resistance of theload (which in turn affects the turbine rotational speed), andby changing the blade pitch angle, we seek to generate thelargest amount of power possible from the turbine. The task
is made challenging by the fact that we do not assume aknown model of the plant dynamics and because we canonly obtain noisy estimates of the power production for agiven set of operating conditions. To accomplish this task
in an efficient manner we develop a new method for policy
search, similar to the REINFORCE algorithm [7] mentioned
above.
To formalize the method, we let w ∈ Rn denote a
set of policy parameters (e.g., the blade pitch and load
resistance for the turbine), and let p(J|w) : R × Rn → R+
denote a probability distribution over costs for a given set ofparameters. The goal is to choose parameters that minimize
the expected cost
Z
E[J|w] = p(J|w)JdJ.
(1)
The difficulty of trying to optimize parameters w is that wecannot directly compute the gradient of the expected costwith respect to w, and indeed this derivative may not exist
depending on the form of p(J|w).
Instead likelihood ratio methods typically consider
stochastic policies that actually maintain a distribution overpossible parameters and optimize with respect to this distribution. In particular, we let θ denote a set of distribution
parameters, and let p(w; θ) denote the distribution overpolicy parameters w given θ. Then the expected cost forparameters θ isZ
E[J; θ] = p(w; θ)E[J|w]dw(2)
and we can represent the gradient of this expected cost asZ∇θ E[J; θ] = ∇θ p(w; θ)E[J|w]dw
Z∇θ p(w; θ)E[J|w]dw= p(w; θ)p(w; θ)(3)Z= p(w; θ)∇θ log p(w; θ)E[J|w]dw= E[J∇θ log p(w; θ)].
The key here is that this last expression involves only
easily computable quantities (since p(w; θ) is known, the∇θ log p(w; θ) term can be computed exactly). Thus thegradient can be approximated by sampling: if we sample
parameters w(1) , . . . , w(m) ∼ p(w; θ), execute these policies
and receive costs J1 , . . . , Jn , then we can approximate thegradient asmX˜ θ E[J; θ] = 1
Ji ∇θ log p(w(i) ; θ)
∇
m i=1

(4)

For the Gaussian noise above, ∇θ log p(w; θ) = ǫ(w − θ)so that the update is equivalent to the so-called weight
perturbation method [28]. Finally, since the gradient is notaffected by adding an arbitrary constant to the cost, it iscommon to change the observed costs to be J˜i = Ji − b for
b chosen to minimize the variance of this estimate [25].
Despite its simplicity, the REINFORCE algorithm hassignificant drawbacks: it often takes several executions of thepolicy in order to obtain a gradient estimate, and even thenone is typically limited to first-order optimization techniquessuch as stochastic gradient descent. Instead, we propose analternative approach that 1) explicitly models second orderfunction information to take larger steps in parameter spaceand 2) makes efficient use of past data so as to quickly
obtain accurate estimates. Formally, we seek a second orderapproximation of E[J|w] that is as accurate as possible forw ∼ p(w; θ), and which captures the change in J withrespect to changes in the gradient of the log probability∇θ log p(w; θ),Ê[J|w] ≈ J0 + g T (∇θ log p(w; θ))(5)1+ (∇θ log p(w; θ))T H(∇θ log p(w; θ)).
2
For p(w; θ) = N (θ, ǫI) (the case we will focus on for theremainder of the paper) this simplifies to a weighted versionof the standard second-order approximation
ǫ2(w − θ)T H(w − θ). (6)2We can find the parameters that maximize the likelihoodof this approximation by simple least-squares regression.
Letting svec : Rn×n → Rn(n+1)/2 denote a function thatmaps symmetric n × n matrices into a vector containing all
their upper triangular entries, we want to find J0 g, and H
so as to minimize
2 


T
svec(H) φ(w)
g(7)
E Ê[J|w] −  J0Ê[J|w] ≈ J0 + ǫg T (w − θ) +where the outer expectation is taken with respect to w ∼p(w; θ) and where 2
(ǫ /2) svec((w − θ)(w − θ)T ).
ǫ(w − θ)φ(w) ≡ (8)1This can be approximated by simply sampling several w(i) ∼p(w; θ) and minimizing the empirical least squares error: wedenote this approximate solution as Jˆ0 , ĝ and Ĥ and forsimplicity of notation we will fold the scaling constants ǫand ǫ2 into ĝ and Ĥ.

While the above approach (in the Gaussian case) just isa standard method for approximating E[J|w] locally as aquadratic function, there are two pitfalls when applying thismethod to policy search. First, when optimizing parameters,
the possible non-convexity of our estimated Ĥ is problematic(either due to the non-convexity of E[J|w] itself, or because
we may not have enough samples to accurately estimate
H). To address this issue, we take an approach common
in optimization and apply a trust region solver to updateparameters θ as
θt+1 ← θt + arg

min∆θ T Q∆θ≤11 T∆θ Ĥ∆θ + ĝ T ∆θ.
2(9)where Q ∈ Rn×n defines is a positive semidefinite trust
region. Although this problem is non-convex for general Ĥ,
it is a well-known trust region sub-problem, and can for
example be solved exactly by semidefinite programming [29,Appendix B]. Further, we argue that trust region methods are
particularly well-suited to policy search. Whereas trust region
methods for optimization typically use solve the trust region
subproblem only approximately, because policy execution
typically the most time-consuming portion of the policy
search process, we can solve these subproblems exactly, using the semidefinite method mentioned above. Furthermore,while choosing the trust region Q is a somewhat arbitrary
decision for most trust region methods, for policy search
there is a natural choice; since Q represents the region where
we are “confident” in our quadratic approximation, choosing
Q = ǫ−1 I (the inverse covariance of our distribution overw) will update the parameters to lie within a region that iswell-approximated by our least-squares approximation of H
and g.
The second difficulty with using second-order approximations for policy search is that the number of samples required
to estimate second-order information can be much higher
than that needed to estimate the gradient alone. To overcome
this limitation, we employ importance sampling, as recently
described in the context of policy search [30], in order toefficiently re-use past data to estimate Ĥ and ĝ. In particular,
let θt be the nominal parameters executed at iteration t of
the algorithm, and suppose that at a previous iteration τ < t,when the nominal parameters were θτ , we and we sampledparameters w(i) ∼ p(w; θτ ). Then even though w(i) was not
sampled from p(w; θt ) we can still use it to estimate theparameters at iteration t using importance-sampling, by reweighting it by the correction factor p(w(i) ; θt )/p(w(i) ; θτ ).
A summary of the entire algorithm, dubbed Trust Region
Policy Search (TRPS), is shown in Algorithm IV.
V. ANALYSIS AND RESULTS
Here we present a number experimental results bothdemonstrating our turbine system and evaluating the TRPS
algorithm. The first set of experiments use exhaustive search
to sweep out the power curve for a number of different
operating conditions of the turbine; this establishes the best
possibly performance might might expect from an online

70
7

Tip Speed Ratio

Algorithm 1 Trust Region Policy Search (TRPS)
Input: initial parameters θ0 ∈ Rn , variance ǫ ∈ R+
Repeat until convergence, t = 0, 1, 2, . . .
1) Sample parameters w(t) ∼ N (θt , ǫI). Execute policy
with parameters w(t) and receive cost Jˆt .
2) For i = 0, . . . , t, compute importance weights
 2
exp ǫ2 (w(i) − θt )T (w(i) − θt )p(w(i) ; θt )

si =
=
2
p(w(i) ; θi )exp ǫ2 (w(i) − θi )T (w(i) − θi )

65
60

6.5

55
6
50
5.5
45

3) Compute weighted least-squares solution
svec(H) ← (ΦT SΦ)−1 ΦT S Jˆ
g
J0

5

40
4

6

8

10
12
Pitch Angle

14

16

where

80


φ(w(0) )T


..
Φ=
 , S = diag(s0 , . . . st ).
.
4) Update distribution parameters
1 Tθt+1 ← θt + arg min∆θ Ĥ∆θ + ĝ T ∆θ.
k∆θk2 ≤ǫ 2

75
Tip Speed Ratioφ(w(t) )T

7

70

6

65
60

5

55
4

50
45

3
4

policy search method, though of course is a much more timeconsuming process. The second set of experiments departs
somewhat from the rest of the paper, but is crucial fordemonstrating the capabilities of the turbine platform: here
we demonstrate individual pitch control at high rotor speeds,
using feed-forward compensation based upon a learned dynamics model of the servo actuators. Finally, we evaluate the
TRPS algorithm on the turbine and show that the method
is able to quickly and reliably obtain near-optimal energy
extraction after about two minutes of execution time on thereal turbine.
A. Modeling and Power Output Analysis
Our first set of experiments characterize the power outputof the turbine relative to a simulated model of the system,
under a variety of different operating conditions. Specifically,
we independently varied the (collective) blade pitch angle ofthe turbine from 4 to 17 degrees in 24 equal increments, andthe resistance from 1.8 to 5 Ohms in 14 equal increments,
for a total of 336 different operating conditions. For eachsetting, we let the turbine run for five seconds (to settle tothe steady state power output), and recorded the resulting
power.
In addition, we developed a simulated model of the systemusing the WT Perf aerodynamics simulator [18], an industry
standard tool developed by the National Wind Technology
Center for predicting the the performance of wind turbines
using blade element momentum (BEM) theory [4]. The
simulator takes as input the physical properties of the systemsuch as the number of blades and their shape, the lift and dragcurves of the blade airfoil, and aerodynamic constants such

6

8

10
12
Pitch Angle

14

16

18

Fig. 9. Real (top) and simulated (bottom) power curves for the turbine
under different operating conditions. Power levels indicated by color are in
Watts. Note the different vertical scales of the two plots.

as the density and viscosity of the air. It then computes the
aerodynamic properties of the blades, along with momentumbased properties of the turbine power output as a whole,
to compute steady-state power output for different operating
conditions of the turbine.
The resulting power curves, for both the real and simulated
systems, are shown in Figure 9. The figures show power
output in Watts, indicated by color, as a function of pitch
angle and tip speed ratio, the ratio of the blade’s tip speed
to the incoming wind speed.
Figure 9 also emphasizes both the the benefits and drawbacks of simulation models. Qualitatively, the two power
curves look quite similar: they both have a characteristic“triangle” shape, and the optimal operating conditions are atsimilar points. But the figures also differ in crucial respects:
the optimal tip speed ratio on the real system is significantly
higher than for the simulated system, and the range of nearoptimal points is significantly smaller. Indeed, if one were tosimply choose the operating conditions of the turbine based
upon the simulated model (as is often done in practice), thenwe would be operating at a substantially suboptimal point.
This highlights the usefulness of experimentation on realsystems and the development of learning methods that can
optimize performance without relying on an a priori modelof the system.

15

B. Independent Pitch Control

14

(10)

where q denotes the motor angle, u denotes the control input(corresponding to the desired angle setpoint), k1 , . . . , k4 areparameters of the model. The model is non-linear due to thesign term, which captures the effects of Coulomb friction.
We fit this model to the data using system identification
procedures: we generate data by commanding a sequence of“chirp” commands (sinusoidal inputs and varying frequencies and magnitudes), then minimize the simulation error of
the model (the deviation between the observed and predictedsequences, simulated over the entire sequence of inputs)
using non-linear optimization. We collect this data when theturbine is stationary, so the models do not attempt to capture
any of the aerodynamic forces on the blades; however, since
the gear inertia in the servos is typically much larger than
the aerodynamic torques associated with the the blades, thesemodels still perform well when the turbine is in operation.
Supposing we want to the pitch angles to track someknow function of rotor angle q d = f (θ), we analyticallydifferentiate this function to achieve desired velocity andaccelerationsq̇ d = f ′ (θ)θ̇,q̈ d = f ′′ (θ)θ̇ + f ′ (θ)θ̈(11)

and assume for simplicity that the rotor speed is constantθ̈ = 0. Instead of commanding the desired angle u = q d , weinvert the dynamics model and command the input
u = q d + (q̈ d − k2 q̇ d − k3 sign(q̇ d ) − xk4 )/k1

(12)which is simply a feed-forward compensator to the desired
position, based upon the dynamical system.
Figure 10 shows the pitch tracking performance using
the model-based feed-forward compensation versus simple
PD control. We commanded a desired angle of q d (θ) =12 − 2.9 sin(θ), with the rotor spinning at 525 RPM. The
top plot of Figure 10 shows tracking performance of the

Pitch Angle (deg)Desired Trajectory
13
12
11
10
9
0

1

2

3
4
Rotor Angle (rad)

5

6

5

6

15
Actual Trajectories
14
Pitch Angle (deg)Our second set of experiments demonstrate the feasibilityof independent pitch control (IPC) on the turbine, a crucial
design requirement for the system. However, IPC on a micro
turbine is a challenging task: in order to maintain reasonable
tip speed ratios, micro turbines must rotate much faster thanlarge turbines (the maximum power output from the previous
section occurs at a tip speed ratio of 6.25, corresponding toa rotor speed of 510 RPM on the turbine). IPC requires at aminimum that we be able to vary the blade pitch on the orderof one rotational period, typically synchronized with the absolute rotor angle in order to account for spatial effects suchas wind shear. Thus, we need both high frequency control
and accurate compensation for the servo motor dynamics inorder to accurately track the desired pitch angles.
To achieve accurate independent pitch control, we employ
feedback compensation using a learned dynamics model. The
dynamics of the motors are well-modeled by a second ordersystemq̈ = k1 (q − u) + k2 q̇ + k3 sign(q̇) + k4

Actual Trajectories

Desired Trajectory

13
12
11
10
9
0

1

2

3
4
Rotor Angle (rad)Fig. 10. Individual Pitch Control tracking performance using learned feedforward compensation (top), and PD control alone (bottom).

feed-forward controller: in this case we are able to track thereference trajectory quite accurately even at high speeds, with
an RMSE of 0.26 degrees. In contrast, the bottom plot shows
the performance of a simple PD controller attempting to track
the trajectory; here the settling time of the motor dynamics
cause the actual trajectory to lag significantly behind thedesired trajectory, a well-known effect in PD control. While
it would also be possible to adaptively tune the phase andamplitude of the desired trajectory to make the PD controller
match the desired behavior, the froward controller does thisautomatically through its model of the dynamics.
C. Policy Search for Online Parameter Optimization
Finally, our last experiments evaluate our proposed trust
region policy search method on the task of optimizating
power output (using collective pitch) at below the turbine’s
rated speed. The policy parameters w ∈ R2 here are simply
the collective blade pitch angle and resistance of the load(since the resistance is more directly controllable, we use it as
our policy parameters, but could equivalently parametrize thepolicy in terms of tip speed ratio). To evaluate the method,
we randomly initialized policies with uniformly random pitch
and resistance values within 10 to 20 degrees, and resistance
between 5 and 0.33 ohms. We then ran our trust region policy
search approach for 40 iterations, using the power output
averaged over two seconds (after three seconds of settlingtime) as the reward signal (the negative cost function).
Figure 11 shows the evolution of the reward versus
iteration number, averaged over 10 trials (with different
starting parameters), along with 95% confidence intervals.
Also shown is the average reward obtained by trusting theWT Perf simulation model and using its prescribed optimal

and uses past data efficiently. We applied the algorithm topower optimization in Region II wind speeds, and showthat it is able to quickly and reliably achieve near-optimal
power output on our turbine. Next steps for the research
involve integrating the policy search and individual pitch
control mechanisms to cope with static obstructions in theincoming airflow. Building upon this, the eventual goal is todevelop a system that can autonomously adjust to dynamic
disturbances, such as those caused by an upwind turbine,using a combination of both model-based and model-free
optimization techniques.

75
70

Average Power (W)

65
60
55
50
45
40
Trust Region Policy Search

35

Optimal Parameters from Model
30

REINFORCE

25
0

5

10

15

20
Iteration

25

30

35

40

Fig. 11. Average reward versus iteration number, with 95% confidence
intervals for our trust region policy search method optimizing power output.

Tip Speed Ratio

7

ACKNOWLEDGMENTS
We thank Morgan Quigley for helpful discussions, Paul
Stearns of Alternate Power Technologies, Inc for the Extractor turbine and his assistance in modifying hub partsfor our turbine, and Ron Wiken for his help designing andbuilding the wind tunnel. J. Zico Kolter is supported by an
NSF Computing Innovations Fellowship.
R EFERENCES

6.5

6

5.5

5
4

6

8

10
12
14
Pitch Angle

16

18

20

Fig. 12. Evolution of policy parameters for a typical run of policy search.

operating point. Despite receiving very little feedback fromthe system, just the average power for a small number ofsituations without any model of the system, our algorithm
is able to quickly obtain near-optimal parameter settings,
typically within about 15 iterations (75 seconds of realtime operation). Figure 12 shows the evolution of the policy
parameters for a typical run of the policy search; as expected
the method quickly gets to (and remains in) a near-optimal
region of the parameter space. In contrast the REINFORCE
algorithm (using both the importance sampling technique
discussed in [30] and the optimal baseline technique from[25]), performs significantly worse in this domain.
VI. CONCLUSIONS AND FUTURE WORKThis paper has presented an actuated wind turbine system. We presented the physical system design, illustrated
the power output of the turbine and demonstrated accurate
individual pitch control via feed-forward compensation. We
proposed and evaluated the Trust Region Policy Searchmethod, an online learning algorithm similar to Reinforcement Learning policy search methods, but which explicitly
models second order information about the cost function[1] Various, “Annual energy review 2009,” U.S. Energy Information
Administration, Tech. Rep., 2009.
[2] ——, “Annual energy outlook 2011,” U.S. Energy Information Administration, Tech. Rep., 2010.
[3] U. D. of Energy, “20% wind energy by 2030: Increasing wind energy’s
contribution to u.s. electricty supply,” U.S. Department of Energy,
Tech. Rep., 2008.
[4] J. Manwell, J. McGowan, and A. Roberts, Wind Energy Explained:
Theory, Design, and Application. Wiley, 2009.
[5] K. E. Johnson, L. Fingersh, M. J. Balas, and L. Y. Pao, “Methods forincreasing region 2 power capture on a variable-speed wind turbine,”
Transactions of the American Society of Mechanical Engineers, vol.
126, 2004.
[6] T. Sebastian and M. Lackner, “Characterization of the unsteady aerodynamics of offshore floating wind turbines,” Wind Energy, vol. 14,
2011.
[7] R. J. Williams, “Simple statistical graident-follwoing algorithms forconnectionist reinforcement learning,” Machine Learning, vol. 8,
no. 23, 1992.
[8] P. Glynn, “Likelihood ratio gradient estimation: an overview,” inProceedigns of the 1987 Winter Simulation Conference, 1987.
[9] T. Burton, D. Sharpe, N. Jenkins, and E. Bossanyi, Wind Energy
Handbook. Wiley, 2001.
[10] T.K.Barlas and G. van Kuik, “Review of state of the art in smart rotor
control research for wind turbines,” Progress in Aerospace Sciences,vol. 46, no. 1, pp. 1–27, 2010.
[11] L. Y. Pao and K. E. Johnson, “Control of wind turbines: Approaches,challenges, and recent developments,” IEEE Control Systems Magazine, vol. 31, pp. 44–62, 2011.
[12] K. Pierce, “Control methods for improved energy capture below rated
power,” in Proceedings of the ASME/JSME Joint Fluids Engineering
Conference, 1999.
[13] J. Youqin, Y. Zhongging, and C. Binggang, “A new maximum power
point tracking control scheme for wind generation,” in Proceedingsof the IEEE International Conference on Power Systems Technology,
2002.
[14] E. Koutroulis and K. Kalaitzakis, “Deisng of a maximum power
tracking system for wind-energy conversion applications,” IEEE Transactions on Industrial Electronics, vol. 53, no. 2, pp. 386–394, 2006.
[15] T. Hawkins, W. White, G. Hu, and F. D. Sahneh, “Wind turbine power
capture control with robust estimation,” in Proceedings of the Dynamic
Systems and Control Conference, 2010.
[16] J. Creaby, Y. Li, and J. Seem, “Maximizing wind turbine energy
capture ssing multi-variable extremum seeking control,” Wind Engineering, vol. 33, no. 4, pp. 361–387, 2009.
[17] “Nwtc design codes (fast by jason jonkman, ph.d.),”http://wind.nrel.gov/designcodes/simulators/fast/.
Last
modified
05-November-2010; accessed 05-November-2010.

[18] “Nwtc
designcodes(wt perfbymarshallbuhl),”
http://wind.nrel.gov/designcodes/simulators/wtperf/. Last modified
17-February-2011; accessed 17-February-2011.
[19] L. Fingersh and K. Johnson, “Controls advanced research turbine(cart) commisioning and baseline data collection,” National Renewable
Energy Laboratory, Tech. Rep., 2002.
[20] A. Wright, L. Fingersh, and M. Balas, “Testing state-space controls forthe controls advanced research turbine,” in Proceedings of the AIAA
Aerospace Sciences Meeting and Exhibit, 2006.
[21] K. A. Stol, W. Zhao, and A. D. Wright, “Individual blade pitch control
for the controls advanced research turbine (cart),” Transactions of the
ASME, vol. 128, pp. 498–505, 2006.
[22] D. Adams, “Model identification and operating load characterization
for a small horizontal axes wind turbine rotor using integrated blade
sensors,” in Proceedings of the Dynamic Systems and Control Conference, 2010.
[23] K. L. Moore, “Iterative learning control: an expository overview,”
Applied and Computational Controls, Signal Processing, and Circuits,
vol. 1, no. 1, pp. 151–214, 1999.
[24] K. Ariyur and M. Krstic, Real-Time Optimization by Extremum-

Seeking Control. Wiley, 2003.
[25] E. Greensmith, P. L. Bartlett, and J. Baxter, “Variance reduction
techniques for gradient estimates in reinforcement learning,” Journalof Machine Learning Research, vol. 5, pp. 1471–1530, 2004.
[26] J. Peters and S. Schaal, “Policy gradient methods for robotics,” inProceedings of the IEEE Conference on Intelligent Robotics Systems,
2006.
[27] A. S. Huang, E. Olson, and D. Moore, “Lcm: Lightweight communications and marshalling,” in Proceedings of the International Conferenceon Intelligent Robots and Systems, 2010.
[28] M. Jabri and B. Flower, “Weight perturbation: an optimal architecture
and learning technique for analog vslif feedforward and recurrent
multilayer networks,” IEEE Transactions and Neural Networks, vol. 3,
pp. 154–147, 1992.
[29] S. Boyd and L. Vandenberghe, Convex Optimization. Cambridge
University Press, 2004.
[30] J. Tang and P. Abbeel, “On a connection between importance sampling and the likelihood ratio policy gradient,” in Neural Information
Processing Systems, 2011.