www.princeton.com

CASE STUDY
OptiSpotter™ Improves High-Frequency Trading ResearchTurnaround Time by 5x-10x While Reducing Costs by 90%
Challenge
Leaders of a High Frequency Trading (HFT) hedge fund retained Princeton Consultants to createa state-of-the-art environment for researching and simulating their alpha and execution ideas.
In the typical cycle, proposed research ideas are discussed, approved, coded, and then backtested against prior market and related data, using different combinations of parameters.
In this never-ending quest, HFT funds face several classic challenges:

One challenge is the sheer quantity of data. For a single research idea, back-testing may
involve reading hundreds of terabytes of data, and require tens of thousands of hours ofprocessing to find and validate the best parameter set.
A second challenge is time. The rate of change in the markets continues to accelerate,
and research turnaround times that used to be acceptable in weeks must now be donein days, even hours.
A third challenge is personnel. Unlike many areas, where vast groups of people can behired and orchestrated, HFT research requires a uniquely skilled and multifaceted
individual, whose time is extremely precious to the overall performance of the fund.
Often these researchers are working massive numbers of hours and still find they can
only investigate a fraction of their ideas effectively.
A final challenge is keeping up with the technology arms race. The drive for ever lower
latencies and ever more nimble algorithms has created an almost insatiable need forcomputation and data storage. Towards this end, it is not unusual for a large financial
institution to purchase and maintain expensive, dedicated supercomputers and farms ofdisk drives for its quantitative research.

Approach & Solution
Princeton Consultants elected to use Amazon Elastic Compute Cloud (Amazon EC2) SpotInstances as a backbone. Spot Instances is a real-time market in which users bid for spare
Amazon EC2 computing capacity.

casestudy.OptiSpotterforHedgeFund.113012© Copyright 2012 Princeton Consultants, Inc. All rights reserved.

Page 1

In an innovative approach, Princeton Consultants refactored the HFT research away frommassive, multi-hour jobs into hundreds of smaller pieces. After evaluating multiple existing
multi-processing controller frameworks, Princeton Consultants created its own, OptiSpotter™.
Using custom optimization algorithms, OptiSpotter™ uses the best mix of servers that is optimalfor the circumstance, in order to complete the current workload in the required time. An
OptiSpotter™ user maps which pieces can run concurrently, and which need to be gated.
OptiSpotter™ marshals these pieces into thousands of small sub-jobs, each with a specificcalculation for a specific region of data.
OptiSpotter™ then places these jobs into SQS queues. The choice of queue depends on thememory and I/O requirements of the sub-job, as well as other factors. For instance, there
might exist several possible instance types in different availability-zones that satisfy the sub-job
requirements, all at different and fluctuating prices.
Using techniques similar to the ones used for the HFT models themselves, the system monitors
the dynamics of the SQS queues, the recent spot price history in several Availability Zones andinstance performance. Using custom optimization algorithms, OptiSpotter™ determines themost efficient collection of spot instances to promptly process the queue contents for the least
cost. In rare cases, the optimal cluster may include some On Demand instances if current spot
prices are highly volatile. Load can be automatically moved to other Regions if cost merits it.
Results
Leveraging the unique advantages of AWS Spot Instances has enabled the hedge fund to greatly
speed up its research, improving turnaround time by 5x-10x. Research ideas that formerly
would have required overnight runs are now available within an hour. This acceleration is a
critical competitive advantage in today’s fast-moving market conditions.
By moving the load from On Demand instances to the price-optimized spot cluster, the fundachieved a 90% reduction in the cost of running the system. The savings meant the fund canresearch ten times as many ideas for the same cost—and provide better and more rigorousalgorithms for its investors.
Finally, the relatively low administrative overhead of an Amazon EC2 cluster, combined with
automatic monitoring of OptiSpotter™ of Spot Instances, saves valuable time for the hedge
fund’s IT personnel. They can focus on the truly proprietary aspects of their trading
infrastructure.
Contact
Steve Sashihara
CEOPrinceton Consultants
SSashihara@princeton.com(609) 987-8787 x214

casestudy.OptiSpotterforHedgeFund.113012© Copyright 2012 Princeton Consultants, Inc. All rights reserved.

Page 2

About Princeton Consultants
Princeton Consultants is an information technology and management consulting firm with
offices in Princeton, NJ and New York, NY. Founded in 1980, the firm has delivered more than
1,000 projects for many of the world’s largest, most successful and innovative companies.

casestudy.OptiSpotterforHedgeFund.113012© Copyright 2012 Princeton Consultants, Inc. All rights reserved.

Page 3