Operations Research — what, when and how | by Aerin Kim | Towards Data Science

The (somewhat vague) term “Operations Research” was coined during World War I. The British military brought together a group of scientists to allocate insufficient resources — for example, food, medics, weapons, troops, etc. — in the most effective way possible to different military operations. So the term “operations” is from “military operations”. Successfully conducting military operations was a huge deal and Operations Research (OR) became its own academic discipline in universities in the 40s.
When you google “Operations Research”, you get a very long Wikipedia article, however, the explanation is a little bit all over the place and to be honest, outdated as well. So I thought I’d give OR, the subject that I studied at graduate school, a nice little update that it deserves.
Here is my take:
Let’s say we are making a decision. If we have to make the best decision possible, what should we do?
You evaluate every possible option by weighing each option’s pros and cons.
For example, in order for Uber to have a master routing plan, it has to decide which driver should be sent where, when, and how much they should charge the customers. And those decisions must be made while optimally using available resources.
I do not know what Uber’s objective function is, but there is something that they are trying to maximize by dispatching the drivers. Let’s say it’s the revenue. There also will be a cost associated with every dispatch and the routing plan should meet the constraints specific to Uber’s policy.
In mathematical terms, the problem above can be written as:
Maximize F(X1, X2, ..., Xn)Such that it meets the constraints C1, C2, ..., Cm.
This type of formulation is called optimization or mathematical programming.
There is an objective function to be maximized (i.e. profit) or minimized (i.e. cost, loss, risk of some undesirable event, etc.) X’s are the decision variables. They are the things we can adjust. For example, each X can be a driver. X_i=1 means the driver i is selected and will be sent to the customer. X_i=0 means he is not selected. C’s are the constraints. For instance, each car has a distance from potential customers. Drivers can drive only for so many hours a day. Each road has a speed limit and each car has a maximum number of passengers it can take.
This is the most common example of Operations Research.
Most of the problems in OR fall into one of the three problem categories.
Mathematical programming: Like the Uber example above, we choose decision variables (drivers to dispatch), the objective function (maximizing the profit) and a set of (physical, technical, economic, environmental, legal, societal, etc.) constraints. Then, we solve them mathematically.
Numerical optimization: Numerical optimization can be either gradient-based or non-gradient. The Gradient Descent, one of the most popular optimization algorithms in Machine Learning, is a gradient-based (obvious from its name) optimization. There are many non-gradient algorithms (derivative-free optimization) as well such as Bayesian optimization, Cuckoo search, Genetic algorithms, etc. Non-gradient algorithms are used when the objective function is not smooth or a closed form of the objective function is not available.
Machine Learning is tightly related to optimization. Many ML problems are formulated as the minimization of some loss function. During the training, the optimization algorithm minimizes the loss on the training set. However, the ultimate goal of ML is to minimize the loss on unseen data. Therefore, Machine Learning is an optimization problem with the goal of "generalization".
A probabilistic model outputs a probability distribution, whereas a deterministic model outputs a single possible outcome for an event.
One of the well-known probabilistic models is cross-entropy, the cost function that we use a lot to predict a probability distribution over the target. Bayesian inference and maximum a-posteriori (MAP) are also important applications of the probabilistic models.
Simulation is used to approximate a probability distribution when deriving one is not handy. It uses repeated random sampling and obtains numerical results. The idea is to use randomness to solve problems that might be deterministic in nature. Simulation has a variety of uses; for example, generating draws from different probability distributions, numerical integration, reinforcement learning, option pricing, etc.
Operations research is applied to a lot of real-world use cases.
Assignment (assigning Uber drivers to customers)
Scheduling (scheduling multiple TV shows together to achieve the maximum views possible)
Financial Engineering (asset allocation, risk management, derivatives pricing, portfolio management, etc.)
Smart Bidding on Youtube, the automated bidding system for algorithmic advertisement (determining how much incremental value can be attributed to a particular impression and how much we should pay for it.)
Pricing Science (airline ticket pricing)
Routing (master planning the routes of buses so that as few buses are needed as possible)
Facility Location (deciding the most appropriate location for new facilities such as warehouse, factory or fire station)
Network Optimization (packet routing)
These OR brain teasers are very frequent in tech interviews, at least in FAAMG. Personally, I was asked two out of three below.
N Queen Problem: How to place N queens on an N x N chessboard so that no two of them attack each other. Two queens should not be on the same row, column, or diagonal.
Traveling Salesman Problem: A classic problem in OR. A salesperson will meet a number of clients in different cities. What’s the shortest round trip? In mathematical terms, given a directed, edge-weighted graph, what is the shortest cyclic path that visits each node in the graph exactly once?
Solution: https://www.geeksforgeeks.org/travelling-salesman-problem-set-1
Stigler Diet: is named after economics Nobel laureate George Stigler, who computed an inexpensive way to fulfill basic nutritional needs given a set of foods. It is a classic linear optimization problem. How do we select a set of foods that will satisfy a set of daily nutritional requirements at minimum cost?
I studied Operations Research at Columbia University for my Master’s degree and there were four required classes to graduate.
You learn about the descriptive statistics, how to construct statistical models, inferential statistics such as finding the maximum likelihood estimator and constructing confidence intervals. You will also learn joint distributions, conditional independence, sums of independent random variables, moment generating functions, laws of large numbers, central limit theorem, infinitely divisible laws, etc.
With a deterministic model, you get the same exact results for particular inputs, no matter how many times you re-run the model. In other words, there is no randomness involved in the deterministic model, which is not very likely in real-world situations. In this class, you’ll learn problem formulation, linear programming, the simplex algorithm, dynamic programming, duality theory, sensitivity theory, etc.
You’ll learn stochastic modeling techniques in Operations Research such as Markov chains, birth and death processes, Poisson processes, gambler’s ruin problem, Brownian motion, etc. For me, this was the most interesting and also challenging class in my program.
You can apply these techniques to reinforcement learning (stochastic processes exploration), financial engineering (pricing call/put/basket options, measuring the insurance risk, etc.), queueing/reliability/inventory modeling, etc.
Unlike deterministic models, the stochastic models take into account the randomness. Therefore, the same model with the same parameters could output different results.
You’ll learn how to generate random variables from different distributions, Monte Carlo, stratified sampling, acceptance-rejection method, variance reduction techniques to make the simulation more efficient, Markov chain Monte Carlo (MCMC), Gibbs sampler, statistical validation techniques to validate the simulation models, how to analyze the simulated output, etc.
There are many electives that you can take as well: Game Theory, Machine Learning (Data Mining), Advanced Optimization, Nonlinear Optimization, Stochastic Control, Real Analysis, Financial Engineering, Asset Allocation, Pricing models, Financial Risk Management, Credit Risk Modeling, Structured and Hybrid Products, Deep Learning, Supply Chain Management, Logistics, etc.
If I track down where my friends are now who used to be in the same program with me, they are everywhere.
Many of them are in the financial industry. Maybe because my school was located close to Wall Street. They work in different teams such as risk management, trading, quantitative research, even in sales or investment banking.
A good number of them became data/applied scientists in tech companies as well. Azure and AWS extensively hire data scientists in order to optimize their cloud downtime, allocate the computing capacity, estimate the real-time demand, etc. Also, the pricing team and the scheduling team at on-demand service companies (Uber, Lyft, AmazonPrimeNow, Doordash, etc.) specifically look for Operations Research majors for their scientist roles. The algorithmic bidding (Adtech) team at social media companies (Twitter, Facebook, YouTube, etc.) has a lot of demands for OR grads for Ad bid pricing, audience targeting, predictive modeling for user engagement, etc.
Personally, I first started working at a financial services company as a Foreign Exchange derivative dealer right after school. Then I joined an adtech startup as a data scientist. Now I’m at Microsoft as a research engineer working on Question Answering (NLP) problem.
If you have a solid grasp of operations research which is applied math, the range of industries that you can choose is very wide.
A few things to add:
According to “No Free Lunch Theorems for Search”, there is no one particular optimization algorithm that works best for every problem.
According to “No Free Lunch Theorems for Search”, there is no one particular optimization algorithm that works best for every problem.
“ We show that all algorithms that search for an extremum of a cost function perform exactly the same, when averaged over all possible cost functions. In particular, if algorithm A outperforms algorithm B on some cost functions, then loosely speaking there must exist exactly as many other functions where B outperforms A.”
Therefore, picking an optimization algorithm that you’ll use should be problem-dependent. For example, when I train deep learning NLP models, my go-to algorithm is ADAM because it works well and it’s fast. I never really use L-BFGS even if it is theoretically faster to converge because based on my experience, SGD is just as good as the second-order algorithms in terms of training time and the final result. However, there will be certain problems where L-BFGS offers better performance than SGD. Here is Quoc Le’s paper (it was published in 2011, so it’s a little old though) “On optimization methods for deep learning” stating that the L-BFGS beats the SGD:
“Our experimental results reflect the different strengths and weaknesses of the different optimization methods. Among the problems we considered, L-BFGS is highly competitive or sometimes superior to SGDs/CG for low dimensional problems, especially convolutional models. For high dimensional problems, CG is more competitive and usually outperforms L-BFGS and SGDs. Additionally, using a large minibatch and line search with SGDs can improve performance”
2. Google has a nice open-source OR-Tools that has solvers for
Constraint Programming
Linear and Mixed-Integer Programming
Bin Packing and Knapsack Algorithms
Algorithms for the Traveling Salesman Problem
Vehicle routing problem
Graph algorithms (shortest paths, min cost flow, max flow, linear sum assignment), etc.