Short notes on Optimization

Optimization is naturally occurring process in many daily, industrial, science and engineering and economics applications. One such natural example is the diet problem where we want to minimize the cost of the food but our nutrient requirements must also be satisfied. Another example is that of a company which wants to maximize its production with constraints on the available resources. Optimization problems also involve shortest path problems where we want to minimize the length of a path between two points provided that there are multiple paths between the two points. Another variant of this is finding a path which takes shortest time between the two points. The shortest path may not give the shortest time due to various constraints (e.g., traffic, quality of road etc.). In control theory, one is interested in optimal actuator/sensor placement problems in large scale networks. In communication engineering, one is interested in optimizing the information transmitted subject to the bandwidth constraints of channels and so on. Fuel and energy optimization problems are common in mechanical and aerospace engineering applications. One is interested in optimizing performance of a control system by minimizing the energy or the fuel required. Some applications involve minimizing the time required e.g., we want to design algorithms which take minimum time possible to solve the problem at hand also minimizing the memory/storage requirements. In electrical engineering, optimization problems occur in energy and power management and also in control and estimation theory and stochastic processes. Design problems are the ones where optimization problems arise naturally. Other applications include computer graphics and animation, risk minimization in investments, device sizing in electronics design, path and motion planning in robotics and so on. There are approximation and estimation problems where we want to minimize the error between approximated/estimated value and the actual value e.g., curve fitting, data fitting, system identification, state estimation in control, image/signal reconstruction in image/signal processing and so on. In applied mathematics, optimization problems occur in least squares problems, norm minimization, approximation of functions by polynomials, approximations by orthogonal projections (e.g. orthogonal projections of vectors, approximations by orthogonal polynomials over an interval and so on), low rank approximation of structured matrices and tensors. Some shortest path problems include finding shortest paths (geodesics) on curved spaces, finding shortest paths on graphs and so on. We now see how to mathematically formulate an optimization problem, identifying the underlying decision variables. We want to understand the theory which allows us to choose these decision variables optimally and solves our problem in principle. Finding the actual solution is not easy in general and requires specific numerical techniques. Understanding optimization problems: The concept of optimization involves an objective to be maximized or minimized by choosing appropriate values of the decision variables. The objective function is a real valued function of the dependent variables. Some variables may not be altered and some may be altered by the designer. Variables which can be altered by the designer are called decision variables. By selecting appropriate values of these decision variables, the designer tries to get the best outcome. (Some decision variables may be redundant as we will see in the next section. These variables may be continuous or discrete. For discrete variables, the problem becomes a combinatorial optimization problem.) The function f : S → R to be minimized or maximized is called the objective function and the underlying set S is called the feasible set. If the feasible set is empty, then no solution exists and