# A New Robust Dynamic Data Envelopment Analysis Approach for Sustainable Supplier Evaluation

Authors: Hava Nikfarjam; Mohsen Rostamy-Malkhalifeh; Abbasali Noura
Journal: Advances in Operations Research
 (2018)

---

## Abstract

Supplier selection is one of the intricate decisions of managers in modern business era. There are different methods and techniques for supplier selection. Data envelopment analysis (DEA) is a popular decision-making method that can be used for this purpose. In this paper, a new dynamic DEA approach is proposed which is capable of evaluating the suppliers in consecutive periods based on their inputs, outputs, and the relationships between the periods classified as desirable relationships, undesirable relationships, and free relationships with positive and negative natures. To this aim various social, economic, and environmental criteria are taken into account. A new method for constructing an ideal decision-making unit (DMU) is proposed in this paper which differs from the existing ones in the literature according to its capability of considering periods with unit efficiencies which do not necessarily belong to a unique DMU. Furthermore, the new ideal DMU has the required ability to rank the suppliers with the same efficiency ratio. In the concerned problem, the supplier that has unit efficiency in each period is selected to construct an ideal supplier. Since it is possible to have more than one supplier with unit efficiency in each period, the ideal supplier can be made with different scenarios with a given probability. To deal with such uncertain condition, a new robust dynamic DEA model is elaborated based on a scenario-based robust optimization approach. Computational results indicate that the proposed robust optimization approach can evaluate and rank the suppliers with unit efficiencies which could not be ranked previously. Furthermore, the proposed ideal DMU can be appropriately used as a benchmark for other DMUs to adjust the probable improvement plans.


---

## Body

## 1. Introduction

Supplier selection is an important strategic decision of managers for the economic and industry. In recent decades scholars and practitioners have paid special attention to this issue. To name a few relevant samples we can refer to Khan et al. [1] who analyzed the suppliers regarding their ability in transferring the technology. However, they merely considered economic criteria in evaluation of four levels of technology transfer among suppliers of auto industries in Pakistan. Nowadays, the decision-maker duty (in supplier selection) has become more and more intricate. This means that they must care specifically about sustainability criteria while supplier selection. Sustainable supplier evaluation and selection concept is resulted from incorporating environmental and social responsibility factors into economic factors when making decisions regarding supply chain management (SCM) [2]. In recent years, sustainability factors have played pivotal role in supplier evaluation and selection process [3]. Ratan et al. [4] discussed that sustainability principles force companies to select the suppliers which develop products and services, preserve environmental resources and look after manpower and communities. Beamon [5] introduced ethical and social responsibilities criteria as fundamental requirements of sustainable SCM for future decades.A wide range of multiple criteria models and approaches such as fuzzy, AHP, ANP, TOPSIS and DEA have been proposed to deal with supplier selection issue over the last two decades. Some of the following researchers applied fuzzy, AHP/ANP-based methods to deal with multi-criteria supplier selection problems (e.g. [6–12]). Some other researchers used TOPSIS based methods to evaluate the suppliers (e.g. [9, 13, 14]). For instance, using fuzzy inference system, Amindoust et al. [15] proposed a ranking model based on fuzzy inference system for sustainable supplier selection. Wen et al. [3] introduced a model for sustainable supplier evaluation using intuitionistic fuzzy sets’ group decision-making models.Another applied method is DEA which is capable to evaluate the suppliers based on weighted ratio of outputs to input. According to Kumar et al. [16], DEA is an applicable and effective tool for supplier selection problem. In the DEA approach, the importance or weights of inputs and outputs are determined through model itself in a pairwise comparison manner without any human’s interference [17]. Different models of DEA have been developed in the literature to evaluate the potential suppliers. For this purpose, Weber et al. (2000) proposed a hybrid multi-objective programming (MOP)-DEA model. Farzipoor Saen [18] proposed a DEA model for ranking suppliers in the presence of imprecise data, weight restriction, and nondiscretionary factors. Also, Farzipoor Saen [19] suggested a DEA model for supplier selection in the presence of undesirable outputs and imprecise data. Noorizadeh et al. [20] introduced a model for supplier selection in the presence of dual-role factors, nondiscretionary inputs, and weight restrictions. To help managers for ranking and selecting the best suppliers in the presence of undesirable outputs and stochastic data, Azadi and Farzipoor Saen [21] developed a new slack- based measure model. Azadi et al. [22] developed a chance-constrained DEA (CCDEA) model for supplier selection in the presence of stochastic data and nondiscretionary factors. Kumar et al. [16] proposed a unified green DEA (GDEA) model for selecting the best suppliers using a comprehensive environment friendly approach.


## 2. Literature Review

Reviewing the relevant literature reveals that traditional models of DEA evaluate efficiency of DMUs merely in one specific past period. Hence, Dynamic DEA (D-DEA) was an appropriate approach which was initially developed by Sengupta [23] and, nowadays, it is used for evaluating DMUs in different periods [24]. In the literature related to dynamic DEA, Färe and Grosspkof [25] proposed a dynamic production frontier using an intermediate output which relates annual production processes. Tone and Tsutsui [26] introduced a new dynamic slack-based measure (DSBM) model to assess DMUs in different periods, using carry-over variables (links). They introduced four types of carry-overs (links) as desirable, undesirable, discretionary, and non-discretionary (fixed) links. Nevertheless, one of the deficiencies of the existing dynamic DEA models is their inability in introducing a strictly efficient DMU with efficiency score of unity in all periods. In fact, strictly efficient DMUs are defined as those that have unit efficiency in all periods. If efficiency score of a DMU in one of the periods has less than unity, it won’t be considered as a strictly efficient unit. This deficiency can be seen in the works by Yousefi et al. [27] and Cook et al. [28]. To overcome this problem, we propose a new method for constructing the ideal DMU (IDMU) where in each period a different DMU with unit efficiency is selected. In fact, the ideal DMU is constructed by a combination of different strictly efficient DMUs. In other words, in this paper, we extend the dynamic DEA model to evaluate suppliers in different periods based on their inputs, outputs and the relationships between the periods classified as desirable relationships, undesirable relationships and free relationships with positive and negative natures. The proposed model is used to evaluate suppliers based on sustainable supplier criteria such as social, economic and environmental criteria.In a methodological point of view, the recent hybrid-approach studies by Tavana et al., 2017; Shabanpour et al., 2017a; Shabanpour et al., 2017b; Yousefi et al., 2016 and Yousefi et al., 2015 have played fundamental roles in creating the main idea and the contributions of this study. Tavana et al., [29] developed a hybrid DEA framework for Sustainable Supplier Evaluation. They combined the goal programming and dynamic DEA model to assess efficiency of suppliers over several periods. Their approach enables the decision-maker to provide improved solutions for inefficient suppliers based on the extent to which the suppliers achieve future goals (benchmarks). Merging dynamic DEA with ANN models, Shabanpour et al., [30] created a novel framework for assessing and forecasting prospective efficiency of green suppliers. Likewise, another relevant survey was conducted by Shabanpour et al., [31]. They applied robust values to define managerial goals (improvement solutions) for evaluating suppliers. Given the fact that the management goals are inherently uncertain as well as based on human interference, they created a new robust double-frontier DEA model to assess and rank sustainable suppliers. Yousefi et al., [32] developed a scenario-based robust DEA technique to deal with sustainable suppliers’ evaluation. Yousefi et al., [33], furthermore, combined goal programming and network DEA and proposed a novel DEA framework to evaluate supply chains. Their approach has the potential of predicting the DMUs’ efficiencies in prospective periods. Accordingly, the decision-maker can not only evaluate suppliers/supply chains but also rank them based on their efficiency trend in several time periods. Tanskanen et al. [34] consider relationship strategies for levels of suppliers with respect to sustainable criteria. Varoutsa and Scapens [35] evaluate the supply chain’s agents inside the organization. Patala et al. [36] consider sustainable criteria such as economic, environmental and social criteria in supplier evaluation.The ideal DMU, as another assessment method, has been used in different performance evaluation problems over the past decade. Wang et al. [37] created an interval DEA model in which efficiency was calculated within the range of an interval. The upper bound of the interval was set to one and the lower bound was established by introducing a virtual IDMU, whose performance was superior to any DMU. Jahanshahloo et al. [38] developed two ranking methods using positive IDMU. They ranked 20 Iranian bank branches by two ranking methods. Hatami-Marbini et al. [39] provided a four-phase fuzzy DEA framework based upon the theory of displaced ideal. They made two hypothetical DMUs namely the ideal and nadir DMUs as reference points to rank the DMUs. Jahanshahloo et al. [40] proposed an interval DEA model to attain an efficiency interval, including evaluations from both the optimistic and the pessimistic perspectives. In their method, the lower bounds of the DMUs are increased to obtain the maximum value. The derived points from this method were called ideal points. Then, the ideal points are employed to rank DMUs. Wang et al. [41] developed new DEA models for cross-efficiency evaluation by introducing a virtual IDMU and a virtual anti-ideal DMU (ADMU). The purpose of their study was to measure the cross-efficiencies in a neutral and more logical way.This paper aims to evaluate the suppliers of a home appliance company based on sustainable suppliers’ criteria using a new robust dynamic DEA model which is capable to evaluate and rank the suppliers with unit efficiencies which could not be ranked in the previously developed approaches. Furthermore, a new method is developed to construct an ideal DMU. The proposed ideal DMU is made up of a combination of DMUs with unit efficiency in each period. It is possible to have more than one DMU with unit efficiency in each period thus resulting in different combinations or scenarios. Therefore, a two-step scenario-based robust approach is employed to deal with these scenarios all with unit efficiencies. The proposed ideal DMU can be appropriately used as a benchmark for other DMUs to adjust the probable improvement plans.The main contributions of this paper are summarized as follows:(i)
Presenting a new method for constructing an ideal DMU in the dynamic DEA model (since it is unlikely to find a DMU with unit efficiency in all periods, in each period a different DMU with unit efficiency can be considered)(ii)
Presenting a two-step robust method to deal with different scenarios for ideal DMU (it is possible to have more than one DMU with unit efficiency in each period and therefore different combinations of DMUs result in different scenarios)(iii)
Presenting robust ranks and improvement plans for all efficient and inefficient unitsCorrespondingly, the following research questions are expected to be addressed in this paper:(i)
What is the efficiency of suppliers in different periods with respect to sustainable criteria?(ii)
What is the rank of suppliers when more than one supplier with unit efficiency exists?(iii)
How can an ideal DMU be constructed to present better improvement methods when no DMU exist with unit efficiency in all periods?(iv)
What if when multiple DMUs have unit efficiency in a period?The rest of the paper is organized as follows. In Section2 the proposed dynamic DEA model is presented; first the deterministic model (Model D) is introduced and then the standard form of the model is written (Model S). Section 3 introduces the proposed two-step robust method which are proposed for the dynamic DEA; For step “a,” the proposed robust input/output-oriented dynamic DEA model is defined (Model RIa/ROa) along with the corresponding linear form (Model LRIa/LROa). Afterwards for step “b,” the proposed robust input/output-oriented dynamic DEA model is defined (Model RIb/ROb) along with the corresponding linear form (Model LRIb/LROb). In Section 4 the proposed robust dynamic DEA models are investigated on a case study for supplier selection. Finally, in Section 5 conclusions are brought along with future research directions.


## 3. Problem Statement and Formulation

The purpose of this paper is to evaluate suppliers of a company in consecutive periods based on sustainable criteria within three categories:(1) social, (2) economic, and (3) environmental. In each period, there are a number of inputs and outputs for each supplier. Also some materials may be transferred from one period to the next period(s) such as backorders and uncashed checks. These make relationships between periods which some of them are desirable and some are undesirable. In the context of DEA, each supplier is considered as a DMU. Since the suppliers are evaluated in multiple periods, dynamic version of DEA is appropriate. Dynamic DEA aims at evaluating m DMUs during P periods with respect to relationships between periods. For every DMU, in each period,n inputs and s outputs are considered. Also, three types of relationships such as desirable, undesirable and free are considered which ensure the link between periods. Relationships with desirable and undesirable nature need to be maximized and minimized, respectively. Free relationships are those that lack any essence and their nature cannot be recognized some of them have positive nature and some have negative nature. Figure 1 graphically illustrates the proposed dynamic DEA.Figure 1
Representation of the proposed dynamic DEA.In the dynamic DEA, usually a strictly efficient unit called ideal DMU is specified to be considered as a benchmark so that inefficient DMUs try to reach it. In fact, the ideal DMU is the one that in all periods has unit efficiency and it is a strictly efficient unit. In most cases such a DMU which is efficient in all periods does not exist and according to the existing definition no ideal DMU can be recognized. To overcome this problem, we introduce a new method for constructing the ideal DMU where in each period a different DMU with unit efficiency in that period is selected. Obviously, the proposed ideal DMU is virtual and does not exist in reality. To clarify, consider 3 DMUs in 5 periods (Figure2). According to the existing method for constructing the ideal DMU, no DMU is selected as an ideal DMU whereas according to the proposed method in this paper, the ideal DMU is composed of DMU 1, DMU 2, DMU 3, DMU 2, and DMU 1, respectively.Figure 2
Demonstrating the proposed ideal DMU.In the dynamic DEA, the efficient frontier is constructed in a pairwise comparison between units where units with maximum ratio of outputs to inputs are selected to construct the efficient frontier as presented by dotted line in Figure3. The improvement method is presented for inefficient units to push them towards this efficient frontier. This issue can be mentioned as another shortcoming of the existing dynamic DEA models where the benchmark(s) as well as improvement plans are merely introduced for inefficient units. Actually, those models do not present improvement methods for benchmarks themselves. The proposed ideal DMU, shown by an asterisk in Figure 3, can be introduced as a benchmark for both inefficient units and efficient units. The boundary which is presented by the solid line in Figure 3 is the frontier which has been constructed by the ideal DMU. As mentioned earlier, the proposed ideal DMU consists of periods with unit efficiencies which do not necessarily belong to a DMU. It is possible to have more than one DMU with unit efficiency in a specific period. Therefore, different scenarios may exist for the ideal DMU. To deal with these scenarios, we employ a scenario-based robust optimization approach and propose a robust dynamic DEA model to present improvement plans for all DMUs. Even when all DMUs obtain similar efficiency, the proposed model can rank those units by considering an absolutely efficient unit called ideal DMU. Actually, the ideal DMU can be used as a unique benchmark based on which improvement methods can be presented for other DMUs. As a matter of fact, initially DMUs are evaluated based on a dynamic DEA approach and then different scenarios are constructed for the ideal DMU (ideal supplier) through a combination of efficient DMUs in each period.Figure 3
Efficient frontiers by the ideal DMU and efficient units.Altogether, the proposed ideal DMU addresses the following concerns about the existing DEA models:(i)
Presenting the improvement methods for efficient units (in existing models the improvement methods can only be presented for inefficient units)(ii)
Considering requirements and opinions of experts in presenting the improvement methods (these opinions are considered in inputs and outputs values of the ideal DMU).(iii)
Modifying the benchmarks in case these units are not acceptable from DM’s perspective.Since in this paper different models are presented, to prevent misunderstanding, we number the models using the following acronyms.Acronyms for ModelsD: Deterministic modelS: Standard modelR I a: Robust Input-oriented model step aR I b: Robust Input-oriented model step bR O a: Robust Output-oriented model step aR O b: Robust Output-oriented model step bL R I a: Linear Robust Input-oriented model step aL R I b: Linear Robust Input-oriented model step bL R O a: Linear Robust Output-oriented model step aL R O b: Linear Robust Output-oriented model step bBefore describing the models, the used notations can be described as follows.Notationsm: Index of DMUsn: Index of inputss: Index of outputsp: Index of time periodsn d, nu,nf: Total number of desirable, undesirable, and free relationships, respectively.x i j p: ith input of thejth DMU in periodp (i= 1, 2, 3, …….,n)y i j p: ith output of thejth DMU in periodp (i= 1, 2, 3, ……,s)z i j p d: Desired relationship for thejth DMU in periodp, (i=1,..,nd), (j=1,…,m), (p=1,…,P).z i j p u: Undesirable relationship for thejth DMU in periodp. (i=1,..,nu), (j=1,…,m), (p=1,…,P).z i j p f: Free relationship for thejth DMU in periodp. (i=1,..,nf), (j=1,…,m), (p=1,…,P).λ j p: Benchmark for thejth inefficient DMU in periodp.o: Index for the under investigation DUMw i -: Weight of the ith inputw i +: Weight of the ith outputw p: Weight of thepth periodϕ o p ∗: Efficiency of the under investigated DMU in periodp (input-oriented model)Φ 0 ∗: Total efficiency of the under investigated DMU (input-oriented model)ψ o p ∗: Efficiency of the under investigated DMU in periodp (output-oriented model)Ψ o ∗: Total efficiency of the under investigated DMU (output-oriented model)


### 3.1. The Proposed Mathematical Model for the Deterministic Dynamic DEA (Model D)

D - 1 - I N max ⁡ Φ 0 ∗ = ∑ p = 1 P ϕ o p ∗ P D - 1 - O U T max ⁡ Ψ o ∗ = ∑ p = 1 P ψ o p ∗ P D - 2 x i o p ≥ ∑ j = 1 m x i j p λ j p  i = 1 , … , n ;   p = 1 , … , P D - 3 y i o p ≤ ∑ j = 1 m y i j p λ j p  i = 1 , … , s ;   p = 1 , … , P D - 4 z i o p d ≤ ∑ j = 1 m z i j p d λ j p  i = 1 , … , n d ;   p = 1 , … , P D - 5 z i o p u ≥ ∑ j = 1 m z i j p u λ j p  i = 1 , … , n u ;   p = 1 , … , P D - 6 z i o p f :  free   o f   s i g n  i = 1 , … , n f ;   p = 1 , … , P D - 7 λ j p ≥ 0  j = 1 , … , m : p = 1 , … , PObjective function D-1-IN and D-1-OUT maximizes the efficiency of the under investigation DMU. D-1-IN is the objective function of the input-oriented model and D-1-OUT is the objective function of the output-oriented model. At each time either objective function D-1-IN or D-1-OUT is considered. Constraint set D-2 ensures that the ith input of the under investigation DMU in period p be greater than or equal to weighted sum of input i in period p for all DMUs. Constraint set D-3 ensures that the ith output of the under investigation DMU in period p be less than or equal to weighted sum of output i in period p for all DMUs. Constraint set D-4 indicates that the value of the ith desirable relationship of the under investigation DMU in period p be less than or equal to weighted sum of the desirable relationship i in period p for all DMUs. Constraint set D-5 indicates that the value of the ith undesirable relationship of the under investigation DMU in period p be greater than or equal to weighted sum of the undesirable relationship i in period p for all DMUs. Constraint set D-6 defines the free relationship variables which are free of sign. Constraint set D-7 defines the weight variables of improvement plans.Note that the right hand sides of the above constraints, i.e.,xijp, yijp,and zijpdzijpu, are positive values. The left hand side of the mentioned constraints, i.e., xiop, yiop, ziopd, ziopu, and ziopf, is connected together through λjp. The continuity of the flow representing the relationship betweenpth and (p+1)th periods is ensured through (1), where ∝ is a general index which can bed,u, or f representing desirable, undesirable, and free relationships. These constraints are important in the proposed dynamic DEA model, since they connect period p to periodp+1 and ensure having a series of time periods. (1)∑j=1mxijp∝λjp=∑j=1mzijp∝λjp+1i=1,…,n∝;p=1,…,P


### 3.2. The Standard Mathematical Model for the Deterministic Dynamic DEA (Model S)

After writing the standard form for the constraints of model (D), the final standard model (S) whose constraints have equal sign is obtained as follows:S-1-INmin⁡Φ0∗=1P∑p=1Pwp1-1m+nu+nf∑i=1nwi-sip-xiop+∑i=1nusipuziopu+∑i=1nfsipf-ziopfS-1-OUTmax⁡1Ψo∗=1P⁡∑p=1Pwp1-1s+nd+nf∑i=1swi+sip+yiop+∑i=1ndsipdziopd+∑i=1nfsipf+ziopfS-2xiop=∑j=1mxijpλjp+sip-i=1,…,n;p=1,…,PS-3yiop=∑j=1myijpλjp-sip+i=1,…,s;p=1,…,PS-4ziopd=∑j=1mzijpdλjp-sipdi=1,…,nd;p=1,…,PS-5ziopu=∑j=1mzijpuλjp+sipui=1,…,nu;p=1,…,PS-6ziopf=∑j=1mzijpfλjp+sipfi=1,…,nf;p=1,…,PS-7sipf:freeofsign∀i,p,sipu≥0,sipp≥0,sip+≥0,sip-≥0,λjp≥0In model (S) like model (D), two alternative cases are considered to calculate the efficiency of the under investigated DMU. One is input-oriented S-1-IN and the other is output-oriented S-1-OUT which are described in the followings. As a matter of fact, the choice of these objective functions depends on the DEA approach, i.e., whether it is input- oriented or output-oriented, which is used for presenting improvement plans.Objective functionS-1-IN represents the total efficiency of the input-oriented model. This objective function is based on the nonredial input-oriented model which considers undesirable relationships, i.e., sipu and sipf-, along with surplus of inputs, i.e., sip-, which should be simultaneously minimized. If all these variables become zero, the efficiency of the considered DMU in period p is one. Obviously, a DMU with overall efficiency equal to one is the one which has unit efficiency in all periods. This objective function calculates the weighted mean of the efficiencies in all periods whose value is between 0 and 1, i.e., (0≤Φo∗≤1), (0≤ϕop∗≤1). The optimal value for the efficiency of period p in input-oriented model is according to (2)ϕop∗=1-1m+nu+nf∑i=1nwi-sip-xiop+∑i=1nusipuziopu+∑i=1nfsipf-ziopf,p=1,…,PObjective function S-1-OUT represents the total efficiency of the output-oriented model. The optimal value for the efficiency of period p in output-oriented model is according to(3)ψop∗=11-1/s+nd+nf∑i=1swi+sip+/yiop+∑i=1ndsipd/ziopd+∑i=1nfsipf+/ziopf,p=1,…,PThe denominator of the objective function deals with the slack of outputs, sip+, free relationships with positive nature, sipf+, and desirable relationships, sipd. If these values become zero, the denominator becomes one and therefore the efficiency of the considered DMU in period p is one. If these slacks get values more than one, the denominator becomes more than one. Therefore, the efficiency of the considered DMU in period p is less than one. Consequently, the total efficiency in the objective function gets a value between zero and one, i.e., (0≤Ψo∗≤1), (0≤ψop∗≤1).In ConstraintS-2, sip- represents the slack for the ith input in period p. The left hand side of this constraint is the inputs of the underinvestigated DMU in period p. If the value of sip- be zero, it means that the supplier does not have excess consumption for that input in period p. In constraint S-3, sip+ represents the surplus for the ith output in period p. In the rest of the constraints, sipd,sipu,sipf, respectively, represent the slack of the desirable relationship, surplus of the undesirable relationship, and the deviation of the free relationship. Note that the auxiliary variables used to standardize constraints have negative natures. For inputs it means excess consumption, for outputs it means shortage in production, for desirable relationships it means shortage in this relationship, for undesirable relationships it means excess in this relationship, and for free relationships it means deviation in this relationship. Constraint S-6 contains a free of sign variable. The deviation of the free relationship can either be stated as slack or surplus. Therefore, to deal with the free of sign variable sipf, two positive variables, sipf- and sipf+, are defined and the following constraints are considered:(4)sipf=sipf--sipf+sipf+∗sipf-=0,sipf-≥0,sipf+≥0Consequently, the following constraints are substituted for Constraint S-6:S-6-1ziopf=∑j=1mzijpfλjp+sipf-i=1,…,nf;p=1,…,PS-6-2ziopf=∑j=1mzijpfλjp-sipf+i=1,…,nf;p=1,…,P


## 3.1. The Proposed Mathematical Model for the Deterministic Dynamic DEA (Model D)

D - 1 - I N max ⁡ Φ 0 ∗ = ∑ p = 1 P ϕ o p ∗ P D - 1 - O U T max ⁡ Ψ o ∗ = ∑ p = 1 P ψ o p ∗ P D - 2 x i o p ≥ ∑ j = 1 m x i j p λ j p  i = 1 , … , n ;   p = 1 , … , P D - 3 y i o p ≤ ∑ j = 1 m y i j p λ j p  i = 1 , … , s ;   p = 1 , … , P D - 4 z i o p d ≤ ∑ j = 1 m z i j p d λ j p  i = 1 , … , n d ;   p = 1 , … , P D - 5 z i o p u ≥ ∑ j = 1 m z i j p u λ j p  i = 1 , … , n u ;   p = 1 , … , P D - 6 z i o p f :  free   o f   s i g n  i = 1 , … , n f ;   p = 1 , … , P D - 7 λ j p ≥ 0  j = 1 , … , m : p = 1 , … , PObjective function D-1-IN and D-1-OUT maximizes the efficiency of the under investigation DMU. D-1-IN is the objective function of the input-oriented model and D-1-OUT is the objective function of the output-oriented model. At each time either objective function D-1-IN or D-1-OUT is considered. Constraint set D-2 ensures that the ith input of the under investigation DMU in period p be greater than or equal to weighted sum of input i in period p for all DMUs. Constraint set D-3 ensures that the ith output of the under investigation DMU in period p be less than or equal to weighted sum of output i in period p for all DMUs. Constraint set D-4 indicates that the value of the ith desirable relationship of the under investigation DMU in period p be less than or equal to weighted sum of the desirable relationship i in period p for all DMUs. Constraint set D-5 indicates that the value of the ith undesirable relationship of the under investigation DMU in period p be greater than or equal to weighted sum of the undesirable relationship i in period p for all DMUs. Constraint set D-6 defines the free relationship variables which are free of sign. Constraint set D-7 defines the weight variables of improvement plans.Note that the right hand sides of the above constraints, i.e.,xijp, yijp,and zijpdzijpu, are positive values. The left hand side of the mentioned constraints, i.e., xiop, yiop, ziopd, ziopu, and ziopf, is connected together through λjp. The continuity of the flow representing the relationship betweenpth and (p+1)th periods is ensured through (1), where ∝ is a general index which can bed,u, or f representing desirable, undesirable, and free relationships. These constraints are important in the proposed dynamic DEA model, since they connect period p to periodp+1 and ensure having a series of time periods. (1)∑j=1mxijp∝λjp=∑j=1mzijp∝λjp+1i=1,…,n∝;p=1,…,P


## 3.2. The Standard Mathematical Model for the Deterministic Dynamic DEA (Model S)

After writing the standard form for the constraints of model (D), the final standard model (S) whose constraints have equal sign is obtained as follows:S-1-INmin⁡Φ0∗=1P∑p=1Pwp1-1m+nu+nf∑i=1nwi-sip-xiop+∑i=1nusipuziopu+∑i=1nfsipf-ziopfS-1-OUTmax⁡1Ψo∗=1P⁡∑p=1Pwp1-1s+nd+nf∑i=1swi+sip+yiop+∑i=1ndsipdziopd+∑i=1nfsipf+ziopfS-2xiop=∑j=1mxijpλjp+sip-i=1,…,n;p=1,…,PS-3yiop=∑j=1myijpλjp-sip+i=1,…,s;p=1,…,PS-4ziopd=∑j=1mzijpdλjp-sipdi=1,…,nd;p=1,…,PS-5ziopu=∑j=1mzijpuλjp+sipui=1,…,nu;p=1,…,PS-6ziopf=∑j=1mzijpfλjp+sipfi=1,…,nf;p=1,…,PS-7sipf:freeofsign∀i,p,sipu≥0,sipp≥0,sip+≥0,sip-≥0,λjp≥0In model (S) like model (D), two alternative cases are considered to calculate the efficiency of the under investigated DMU. One is input-oriented S-1-IN and the other is output-oriented S-1-OUT which are described in the followings. As a matter of fact, the choice of these objective functions depends on the DEA approach, i.e., whether it is input- oriented or output-oriented, which is used for presenting improvement plans.Objective functionS-1-IN represents the total efficiency of the input-oriented model. This objective function is based on the nonredial input-oriented model which considers undesirable relationships, i.e., sipu and sipf-, along with surplus of inputs, i.e., sip-, which should be simultaneously minimized. If all these variables become zero, the efficiency of the considered DMU in period p is one. Obviously, a DMU with overall efficiency equal to one is the one which has unit efficiency in all periods. This objective function calculates the weighted mean of the efficiencies in all periods whose value is between 0 and 1, i.e., (0≤Φo∗≤1), (0≤ϕop∗≤1). The optimal value for the efficiency of period p in input-oriented model is according to (2)ϕop∗=1-1m+nu+nf∑i=1nwi-sip-xiop+∑i=1nusipuziopu+∑i=1nfsipf-ziopf,p=1,…,PObjective function S-1-OUT represents the total efficiency of the output-oriented model. The optimal value for the efficiency of period p in output-oriented model is according to(3)ψop∗=11-1/s+nd+nf∑i=1swi+sip+/yiop+∑i=1ndsipd/ziopd+∑i=1nfsipf+/ziopf,p=1,…,PThe denominator of the objective function deals with the slack of outputs, sip+, free relationships with positive nature, sipf+, and desirable relationships, sipd. If these values become zero, the denominator becomes one and therefore the efficiency of the considered DMU in period p is one. If these slacks get values more than one, the denominator becomes more than one. Therefore, the efficiency of the considered DMU in period p is less than one. Consequently, the total efficiency in the objective function gets a value between zero and one, i.e., (0≤Ψo∗≤1), (0≤ψop∗≤1).In ConstraintS-2, sip- represents the slack for the ith input in period p. The left hand side of this constraint is the inputs of the underinvestigated DMU in period p. If the value of sip- be zero, it means that the supplier does not have excess consumption for that input in period p. In constraint S-3, sip+ represents the surplus for the ith output in period p. In the rest of the constraints, sipd,sipu,sipf, respectively, represent the slack of the desirable relationship, surplus of the undesirable relationship, and the deviation of the free relationship. Note that the auxiliary variables used to standardize constraints have negative natures. For inputs it means excess consumption, for outputs it means shortage in production, for desirable relationships it means shortage in this relationship, for undesirable relationships it means excess in this relationship, and for free relationships it means deviation in this relationship. Constraint S-6 contains a free of sign variable. The deviation of the free relationship can either be stated as slack or surplus. Therefore, to deal with the free of sign variable sipf, two positive variables, sipf- and sipf+, are defined and the following constraints are considered:(4)sipf=sipf--sipf+sipf+∗sipf-=0,sipf-≥0,sipf+≥0Consequently, the following constraints are substituted for Constraint S-6:S-6-1ziopf=∑j=1mzijpfλjp+sipf-i=1,…,nf;p=1,…,PS-6-2ziopf=∑j=1mzijpfλjp-sipf+i=1,…,nf;p=1,…,P


## 4. Robust Dynamic DEA Model

As mentioned previously, one of the deficiencies of the existing dynamic DEA is the lack of a strictly efficient unit as a unit that can be introduced as a benchmark. To overcome this deficiency, we introduce a new method for constructing the ideal DMU which is constructed by making use of the results obtained from the dynamic DEA. The proposed ideal DMU is made up of different periods each of which contains DMUs with unit efficiency. As a matter of fact, in each period, DMUs are evaluated and DMU(s) with unit efficiency are selected to construct the ideal DMU. Since more than one DMU with unit efficiency may exist in each period, different combinations of DMUs may be generated for the ideal DMU. Each of these combinations is called a “scenario”. More than one DMU with unit efficiency in each period leads to different combinations or scenarios for the ideal DMU whose probabilities of occurrence are considered the same in this paper. Different scenarios for the ideal DMU result in different improvement plans. By taking the advantages of the scenario-based robust optimization method and applying it for the studying dynamic DEA, we evaluate and rank the suppliers with respect to these scenarios for the ideal DMU. This process is done in two steps. The first step (step a) formulates the robust optimization model where one of the scenarios is under investigation. The second step (step b) formulates the robust optimization model where other DMUs along with the selected scenario unit are under investigation.The procedure for the proposed supplier evaluation and rank model is summarized in the following procedure.Procedure: Supplier Evaluation and Rank through the Proposed Robust Dynamic DEABegin (1)
Determine inputs, outputs, desirable, and undesirable relationships for suppliers in each period. (2)
Consider each supplier as a DMU and employ the dynamic (input/ouput-oriented) DEA model to determine the efficiency values of each supplier in each period. (3)
If there is a DMU with unit efficiency values in all periods consider it as a strictly efficient unit. (4)
Otherwise, build a virtual ideal DMU whose periods belong to DMUs with unit efficiency thus leading to different scenarios for the ideal DMU. (5)
In step “a,” evaluate and rank scenarios (with equal probability and based on the punishment and encouragement values considered for each scenario) using the proposed linear (input/output-oriented) robust dynamic DEA model (LRIa/LROa). The best scenario is considered as a unique benchmark for presenting improvement plans. (6)
In step “b,” consider the selected scenario from step “a” along with other suppliers (resulting inm+1 number of DMUs) and evaluate the suppliers through models (LRIb/LROb).End.Notations Used in the Proposed Robust MethodП s: The probability of occurrence for scenario sk i s: The unit cost for ith input ofsth scenario in periodp.g i s: The unit cost for ith undesirable relationship ofsth scenario in periodp.v i s: The unit cost for ith free relationship ofsth scenario in periodp.h i s : The unit revenue for ith output ofsth scenario in periodp.b i s : The unit revenue for ith desirable relationship ofsth scenario in periodp.e i s: The unit revenue for ith free relationship with positive nature ofsth scenario in periodp.


### 4.1. Step a: A Scenario Unit Is under Investigation

In this section the efficiency of scenarios are investigated through modelsRIa or ROa, depending on the decision-maker’s approach which could be input-oriented or output-oriented. At each time one of the scenarios is under investigation and the efficiencies of scenarios for the ideal DMU are calculated and they are ranked. The high ranked scenario is selected based on which the improvement plan is presented. As a matter of fact, the best scenario has more distance from other DMUs (see Figure 3) and the improvement plan for other DMUs is presented in the worst case. Therefore, the proposed improvement plan is robust against different scenarios which could be considered for the ideal DMU.


#### 4.1.1. Robust Optimization for Input Oriented DEA Model Step a (RIa)

R I a - 1 min ⁡ β × A v e r a g e + 1 - β × ∑ s = 1 S П s × 1 P ⁡ ∑ p = 1 P w p 1 1 - 1 / m + n u + n f ∑ i = 1 n w i - s i p - / x i o p + ∑ i = 1 n u s i p u / z i o p u + ∑ i = 1 n f s i p f - / z i o p f - A v e r a g e + ∑ s = 1 S П s ∑ i = 1 n k i s x s i s p λ S s p + ∑ i = 1 n u g i s z s i s p u λ s s p + ∑ i = 1 n f v i s z s i s p f - λ s s p - ∑ i = 1 s h i s y s i s p λ s s p - ∑ i = 1 n d b i s z s i s p d λ s s p - ∑ i = 1 n f e i s z s i s p f + λ s s p R I a - 2 A v e r a g e = ∑ s = 1 S П s × 1 P ⁡ ∑ p = 1 P w p 1 - 1 m + n u + n f ∑ i = 1 n w i - s i s p - x i o p + ∑ i = 1 n u s i s p u z i o p u + ∑ i = 1 n f s i s p f - z i o p f R I a - 3 x i s p = ∑ j = 1 m x i j p λ j p + x s i s p λ S s p + s i s p -  i = 1 , … , n ;   p = 1 , … , P ;   s = 1 , … , S R I a - 4 y s i s p = ∑ j = 1 m y i j p λ j p + y s i s p λ s s p - s i s p +  i = 1 , … , s ;   p = 1 , … , P ;   s = 1 , … , S R I a - 5 z s i o p f = ∑ j = 1 m z i j p f λ j p + z s i s p f λ s s p + s i s p f -  i = 1 , … , n f ;   p = 1 , … , P R I a - 6 z s i o p f = ∑ j = 1 m z i j p f λ j p + z s i s p f λ s s p - s i s p f +  i = 1 , … , n f ;   p = 1 , … , P R I a - 7 z s i o p d = ∑ j = 1 m z i j p d λ j p + z s i s p d λ s s p - s i s p d  i = 1 , … , n d ;   p = 1 , … , P R I a - 8 z s i s p u = ∑ j = 1 m z i j p u λ j p + z s i s p u λ s s p + s i s p u  i = 1 , … , n u ;   p = 1 , … , P ;   s = 1 , … , S R I a - 9 s i s p u ≥ 0 , s i p s + ≥ 0 , s i s p - ≥ 0 , λ j p ≥ 0 , s s p ≥ 0 , s i s p f + ≥ 0 , s i s p f - ≥ 0 , s i s p d ≥ 0The objective function RIa-1 consists of three terms. The first term calculates the average efficiency of scenarios. The second term calculates the deviation of efficiency of scenarios from the average value. The third term calculates the total profit or loss resulted from scenarios. In fact, in each scenario, outputs, desirable relationships and free relationships with positive natures, yield return. Whilst inputs, undesirable relationships, and free relationships with negative natures result in cost. Note that in this term, the values of returns are subtracted from the values of costs. Therefore, if this term is negative it means that the considering scenario is profitable; otherwise it makes losses.It is worth noting that since each scenario is a combination of DMUs with unit efficiency selected from different periods, the efficiency of each scenario is one. Therefore, the average efficiency of all scenarios is one and consequently the standard deviation is zero.(1) Linear Robust Input-Oriented Model Step a ( L RI a ). To deal with the absolute function and make the model linear, two positive variables Qs+ and Qs- are defined. LRIa-1minΦ0∗=β×Average+1-β×∑s=1SПs×Qs++Qs-+∑s=1SПs∑i=1nkisxsispλSsp+∑i=1nugiszsispuλssp+∑i=1nfviszsispf-λssp-∑i=1shisysispλssp-∑i=1ndbiszsispdλssp-∑i=1nfeiszsispf+λsspLRIa-21P⁡∑p=1Pwp1-1m+nu+nf∑i=1nwi-sisp-xiop+∑i=1nusispuziopu+∑i=1nfsispf-ziopf-Average=Qs+-Qs-Other constraints of model (RIa) hold.


#### 4.1.2. Robust Optimization for Output-Oriented DEA Model Step a(ROa)

The robust optimization for the output-oriented model differs with the input oriented model in the objective function and also in the average and the linearized constraints. Other constraints are the same for both.ROa-1min⁡β×Average+1-β×∑s=1SПs×1P∑p=1Pwp1-1s+nd+nf∑i=1swi+sip+yiop+∑i=1ndsipdziopd+∑i=1nfsipf+ziopf-Average+∑s=1SПs∑i=1nkisxsispλSsp+∑i=1nugiszsispuλssp+∑i=1nfviszsispf-λssp-∑i=1shisysispλssp-∑i=1ndbiszsispdλssp-∑i=1nfeiszsispf+λsspROa-2Average=∑s=1SПs×1P∑p=1Pwp1-1s+nd+nf∑i=1swi+sip+yiop+∑i=1ndsipdziopd+∑i=1nfsipf+ziopfOther constraints of model (RIa) hold.Like the input-oriented model, the objective functionROa-1 consists of three terms. The first term minimizes the average efficiency of scenarios with weight importance of β. The second term minimizes the standard deviation of efficiencies with weight importance of 1- β. Finally, the third term calculates the total profit or loss resulted from scenarios.(1) Linear Robust Output-Oriented Model Step a ( L R O a ). By considering two positive variables Qs+ and Qs- and substituting it with the absolute function, the model is linearized. LROa-1min⁡β×Average+1-β×∑s=1SПs×Qs++Qs-+∑s=1SПs∑i=1nkisxsispλSsp+∑i=1nugiszsispuλssp+∑i=1nfviszsispf-λssp-∑i=1shisysispλssp-∑i=1ndbiszsispdλssp-∑i=1nfeiszsispf+λsspLROa-21P∑p=1Pwp1-1s+nd+nf∑i=1swi+sip+yiop+∑i=1ndsipdziopd+∑i=1nfsipf+ziopf-Average=Qs+-Qs-Other constraints of model (RIa) hold.


### 4.2. Step b: A DMU from Other DMUs Is under Investigation

In the previous section the under investigation DMU was one of the scenarios and we evaluated scenarios and selected the suitable one. In fact, model (RIa/ROa) calculates benefit or loss resulted from each scenario and we can select the best one accordingly. In this section, other DMUs are evaluated along with the selected scenario and therefore the number of DMUs increases by one (i.e.,m+1). Actually the best scenario is considered in model (RIb/ROb).


#### 4.2.1. Robust Optimization for Input-Oriented DEA Model Step b (RIb)

The following model (RIb) evaluates other DMUs along with the strictly efficient DMU (i.e., the selected scenario as an ideal DMU) as a benchmark. Then the improvement methods can be presented for other DMUs based on their distance from the efficient frontier and image inefficient DMUs to the efficient frontier. The main difference of model (RIb) with model (RIa) is that in model (RIb) the number of DMUs is m+1. Another difference is that the objective function consists of two terms, i.e., the average efficiency and the standard deviation of efficiencies of the considered DMU in different periods respectively with importance weights β and 1-β. Note that the objective function of model (RIb) does not consider the cost since by taking the ideal DMU into consideration; we can rank DMUs and present improvement methods.RIb-1min⁡Φ0∗=β×Average+1-β×∑s=1SПs×1P⁡∑p=1Pwp1-1m+nu+nf∑i=1nwi-sip-xiop+∑i=1nusipuziopu+∑i=1nfsipf-ziopf-AverageRIb-2s.t.Average=∑s=1SПs×1P⁡∑p=1Pwp1-1m+nu+nf∑i=1nwi-sip-xiop+∑i=1nusipuziopu+∑i=1nfsipf-ziopfRIb-3xiop=∑j=1m+1xijpλjp+sip-i=1,…,n;p=1,…,P;j=1,…,m+1RIb-4yiop=∑j=1m+1yijpλjp-sip+i=1,…,s;p=1,…,P;j=1,…,m+1RIb-5ziopu=∑j=1m+1zijpuλjp+sipui=1,…,nu;p=1,…,P;j=1,…,m+1RIb-6zsiopf=∑j=1m+1zijpfλjp+sipf-i=1,…,nf;p=1,…,P;j=1,…,m+1RIb-7zsiopf=∑j=1mzijpfλjp-sipf+i=1,…,nf;p=1,…,P;j=1,…,m+1RIb-8zsiopd=∑j=1mzijpdλjp-sipdi=1,…,nd;p=1,…,P;j=1,…,m+1RIb-9sipu≥0,sip+≥0,sip-≥0,λjp≥0,sipf+≥0,sipf-≥0,sipd≥0(1) Linear Robust Input-Oriented Model Step b ( L R I b ). By considering two positive variables Qs+and Qs- and substituting it with the absolute function, the model is linearized. LRIb-1minΦ0∗=β×Average+1-β×∑s=1SПs×Qs++Qs-LRIb-2s.t.Average=∑s=1SПs×1P⁡∑p=1Pwp1-1m+nu+nf∑i=1nwi-sip-xiop+∑i=1nusipuziopu+∑i=1nfsipf-ziopfLRIb-31P⁡∑p=1Pwp1-1m+nu+nf∑i=1nwi-sip-xiop+∑i=1nusipuziopu+∑i=1nfsipf-ziopf-Average=Qs+-Qs-Other constraints of model (RIb) hold.


#### 4.2.2. Robust Optimization for Output-Oriented DEA Model Step b (ROb)

R O b - 1 m i n  Φ 0 ∗ = β × A v e r a g e + 1 - β Φ 0 ∗ = × ∑ s = 1 S П s × 1 P ∑ p = 1 P w p 1 - 1 s + n d + n f ∑ i = 1 s w i + s i p + y i o p + ∑ i = 1 n d s i p d z i o p d + ∑ i = 1 n f s i p f + z i o p f - A v e r a g e R O b - 2 s . t .  A v e r a g e = ∑ s = 1 S П s × 1 P ∑ p = 1 P w p 1 - 1 s + n d + n f ∑ i = 1 s w i + s i p + y i o p + ∑ i = 1 n d s i p d z i o p d + ∑ i = 1 n f s i p f + z i o p fOther constraints of model (RIb) hold.(1) Linear Robust Output-Oriented Model Step b ( L R O b ). By considering two positive variables Qs+ and Qs- and substituting them with the absolute function, the model is linearized. LROb-1minΦ0∗=β×Average+1-β×∑s=1SПs×Qs++Qs-LROb-2s.t.Average=∑s=1SПs×1P∑p=1Pwp1-1s+nd+nf∑i=1swi+sip+yiop+∑i=1ndsipdziopd+∑i=1nfsipf+ziopfLROb-31P∑p=1Pwp1-1s+nd+nf∑i=1swi+sip+yiop+∑i=1ndsipdziopd+∑i=1nfsipf+ziopf-Average=Qs+-Qs-Other constraints of model (RIb) hold.


## 4.1. Step a: A Scenario Unit Is under Investigation

In this section the efficiency of scenarios are investigated through modelsRIa or ROa, depending on the decision-maker’s approach which could be input-oriented or output-oriented. At each time one of the scenarios is under investigation and the efficiencies of scenarios for the ideal DMU are calculated and they are ranked. The high ranked scenario is selected based on which the improvement plan is presented. As a matter of fact, the best scenario has more distance from other DMUs (see Figure 3) and the improvement plan for other DMUs is presented in the worst case. Therefore, the proposed improvement plan is robust against different scenarios which could be considered for the ideal DMU.


### 4.1.1. Robust Optimization for Input Oriented DEA Model Step a (RIa)

R I a - 1 min ⁡ β × A v e r a g e + 1 - β × ∑ s = 1 S П s × 1 P ⁡ ∑ p = 1 P w p 1 1 - 1 / m + n u + n f ∑ i = 1 n w i - s i p - / x i o p + ∑ i = 1 n u s i p u / z i o p u + ∑ i = 1 n f s i p f - / z i o p f - A v e r a g e + ∑ s = 1 S П s ∑ i = 1 n k i s x s i s p λ S s p + ∑ i = 1 n u g i s z s i s p u λ s s p + ∑ i = 1 n f v i s z s i s p f - λ s s p - ∑ i = 1 s h i s y s i s p λ s s p - ∑ i = 1 n d b i s z s i s p d λ s s p - ∑ i = 1 n f e i s z s i s p f + λ s s p R I a - 2 A v e r a g e = ∑ s = 1 S П s × 1 P ⁡ ∑ p = 1 P w p 1 - 1 m + n u + n f ∑ i = 1 n w i - s i s p - x i o p + ∑ i = 1 n u s i s p u z i o p u + ∑ i = 1 n f s i s p f - z i o p f R I a - 3 x i s p = ∑ j = 1 m x i j p λ j p + x s i s p λ S s p + s i s p -  i = 1 , … , n ;   p = 1 , … , P ;   s = 1 , … , S R I a - 4 y s i s p = ∑ j = 1 m y i j p λ j p + y s i s p λ s s p - s i s p +  i = 1 , … , s ;   p = 1 , … , P ;   s = 1 , … , S R I a - 5 z s i o p f = ∑ j = 1 m z i j p f λ j p + z s i s p f λ s s p + s i s p f -  i = 1 , … , n f ;   p = 1 , … , P R I a - 6 z s i o p f = ∑ j = 1 m z i j p f λ j p + z s i s p f λ s s p - s i s p f +  i = 1 , … , n f ;   p = 1 , … , P R I a - 7 z s i o p d = ∑ j = 1 m z i j p d λ j p + z s i s p d λ s s p - s i s p d  i = 1 , … , n d ;   p = 1 , … , P R I a - 8 z s i s p u = ∑ j = 1 m z i j p u λ j p + z s i s p u λ s s p + s i s p u  i = 1 , … , n u ;   p = 1 , … , P ;   s = 1 , … , S R I a - 9 s i s p u ≥ 0 , s i p s + ≥ 0 , s i s p - ≥ 0 , λ j p ≥ 0 , s s p ≥ 0 , s i s p f + ≥ 0 , s i s p f - ≥ 0 , s i s p d ≥ 0The objective function RIa-1 consists of three terms. The first term calculates the average efficiency of scenarios. The second term calculates the deviation of efficiency of scenarios from the average value. The third term calculates the total profit or loss resulted from scenarios. In fact, in each scenario, outputs, desirable relationships and free relationships with positive natures, yield return. Whilst inputs, undesirable relationships, and free relationships with negative natures result in cost. Note that in this term, the values of returns are subtracted from the values of costs. Therefore, if this term is negative it means that the considering scenario is profitable; otherwise it makes losses.It is worth noting that since each scenario is a combination of DMUs with unit efficiency selected from different periods, the efficiency of each scenario is one. Therefore, the average efficiency of all scenarios is one and consequently the standard deviation is zero.(1) Linear Robust Input-Oriented Model Step a ( L RI a ). To deal with the absolute function and make the model linear, two positive variables Qs+ and Qs- are defined. LRIa-1minΦ0∗=β×Average+1-β×∑s=1SПs×Qs++Qs-+∑s=1SПs∑i=1nkisxsispλSsp+∑i=1nugiszsispuλssp+∑i=1nfviszsispf-λssp-∑i=1shisysispλssp-∑i=1ndbiszsispdλssp-∑i=1nfeiszsispf+λsspLRIa-21P⁡∑p=1Pwp1-1m+nu+nf∑i=1nwi-sisp-xiop+∑i=1nusispuziopu+∑i=1nfsispf-ziopf-Average=Qs+-Qs-Other constraints of model (RIa) hold.


### 4.1.2. Robust Optimization for Output-Oriented DEA Model Step a(ROa)

The robust optimization for the output-oriented model differs with the input oriented model in the objective function and also in the average and the linearized constraints. Other constraints are the same for both.ROa-1min⁡β×Average+1-β×∑s=1SПs×1P∑p=1Pwp1-1s+nd+nf∑i=1swi+sip+yiop+∑i=1ndsipdziopd+∑i=1nfsipf+ziopf-Average+∑s=1SПs∑i=1nkisxsispλSsp+∑i=1nugiszsispuλssp+∑i=1nfviszsispf-λssp-∑i=1shisysispλssp-∑i=1ndbiszsispdλssp-∑i=1nfeiszsispf+λsspROa-2Average=∑s=1SПs×1P∑p=1Pwp1-1s+nd+nf∑i=1swi+sip+yiop+∑i=1ndsipdziopd+∑i=1nfsipf+ziopfOther constraints of model (RIa) hold.Like the input-oriented model, the objective functionROa-1 consists of three terms. The first term minimizes the average efficiency of scenarios with weight importance of β. The second term minimizes the standard deviation of efficiencies with weight importance of 1- β. Finally, the third term calculates the total profit or loss resulted from scenarios.(1) Linear Robust Output-Oriented Model Step a ( L R O a ). By considering two positive variables Qs+ and Qs- and substituting it with the absolute function, the model is linearized. LROa-1min⁡β×Average+1-β×∑s=1SПs×Qs++Qs-+∑s=1SПs∑i=1nkisxsispλSsp+∑i=1nugiszsispuλssp+∑i=1nfviszsispf-λssp-∑i=1shisysispλssp-∑i=1ndbiszsispdλssp-∑i=1nfeiszsispf+λsspLROa-21P∑p=1Pwp1-1s+nd+nf∑i=1swi+sip+yiop+∑i=1ndsipdziopd+∑i=1nfsipf+ziopf-Average=Qs+-Qs-Other constraints of model (RIa) hold.


## 4.1.1. Robust Optimization for Input Oriented DEA Model Step a (RIa)

R I a - 1 min ⁡ β × A v e r a g e + 1 - β × ∑ s = 1 S П s × 1 P ⁡ ∑ p = 1 P w p 1 1 - 1 / m + n u + n f ∑ i = 1 n w i - s i p - / x i o p + ∑ i = 1 n u s i p u / z i o p u + ∑ i = 1 n f s i p f - / z i o p f - A v e r a g e + ∑ s = 1 S П s ∑ i = 1 n k i s x s i s p λ S s p + ∑ i = 1 n u g i s z s i s p u λ s s p + ∑ i = 1 n f v i s z s i s p f - λ s s p - ∑ i = 1 s h i s y s i s p λ s s p - ∑ i = 1 n d b i s z s i s p d λ s s p - ∑ i = 1 n f e i s z s i s p f + λ s s p R I a - 2 A v e r a g e = ∑ s = 1 S П s × 1 P ⁡ ∑ p = 1 P w p 1 - 1 m + n u + n f ∑ i = 1 n w i - s i s p - x i o p + ∑ i = 1 n u s i s p u z i o p u + ∑ i = 1 n f s i s p f - z i o p f R I a - 3 x i s p = ∑ j = 1 m x i j p λ j p + x s i s p λ S s p + s i s p -  i = 1 , … , n ;   p = 1 , … , P ;   s = 1 , … , S R I a - 4 y s i s p = ∑ j = 1 m y i j p λ j p + y s i s p λ s s p - s i s p +  i = 1 , … , s ;   p = 1 , … , P ;   s = 1 , … , S R I a - 5 z s i o p f = ∑ j = 1 m z i j p f λ j p + z s i s p f λ s s p + s i s p f -  i = 1 , … , n f ;   p = 1 , … , P R I a - 6 z s i o p f = ∑ j = 1 m z i j p f λ j p + z s i s p f λ s s p - s i s p f +  i = 1 , … , n f ;   p = 1 , … , P R I a - 7 z s i o p d = ∑ j = 1 m z i j p d λ j p + z s i s p d λ s s p - s i s p d  i = 1 , … , n d ;   p = 1 , … , P R I a - 8 z s i s p u = ∑ j = 1 m z i j p u λ j p + z s i s p u λ s s p + s i s p u  i = 1 , … , n u ;   p = 1 , … , P ;   s = 1 , … , S R I a - 9 s i s p u ≥ 0 , s i p s + ≥ 0 , s i s p - ≥ 0 , λ j p ≥ 0 , s s p ≥ 0 , s i s p f + ≥ 0 , s i s p f - ≥ 0 , s i s p d ≥ 0The objective function RIa-1 consists of three terms. The first term calculates the average efficiency of scenarios. The second term calculates the deviation of efficiency of scenarios from the average value. The third term calculates the total profit or loss resulted from scenarios. In fact, in each scenario, outputs, desirable relationships and free relationships with positive natures, yield return. Whilst inputs, undesirable relationships, and free relationships with negative natures result in cost. Note that in this term, the values of returns are subtracted from the values of costs. Therefore, if this term is negative it means that the considering scenario is profitable; otherwise it makes losses.It is worth noting that since each scenario is a combination of DMUs with unit efficiency selected from different periods, the efficiency of each scenario is one. Therefore, the average efficiency of all scenarios is one and consequently the standard deviation is zero.(1) Linear Robust Input-Oriented Model Step a ( L RI a ). To deal with the absolute function and make the model linear, two positive variables Qs+ and Qs- are defined. LRIa-1minΦ0∗=β×Average+1-β×∑s=1SПs×Qs++Qs-+∑s=1SПs∑i=1nkisxsispλSsp+∑i=1nugiszsispuλssp+∑i=1nfviszsispf-λssp-∑i=1shisysispλssp-∑i=1ndbiszsispdλssp-∑i=1nfeiszsispf+λsspLRIa-21P⁡∑p=1Pwp1-1m+nu+nf∑i=1nwi-sisp-xiop+∑i=1nusispuziopu+∑i=1nfsispf-ziopf-Average=Qs+-Qs-Other constraints of model (RIa) hold.


## 4.1.2. Robust Optimization for Output-Oriented DEA Model Step a(ROa)

The robust optimization for the output-oriented model differs with the input oriented model in the objective function and also in the average and the linearized constraints. Other constraints are the same for both.ROa-1min⁡β×Average+1-β×∑s=1SПs×1P∑p=1Pwp1-1s+nd+nf∑i=1swi+sip+yiop+∑i=1ndsipdziopd+∑i=1nfsipf+ziopf-Average+∑s=1SПs∑i=1nkisxsispλSsp+∑i=1nugiszsispuλssp+∑i=1nfviszsispf-λssp-∑i=1shisysispλssp-∑i=1ndbiszsispdλssp-∑i=1nfeiszsispf+λsspROa-2Average=∑s=1SПs×1P∑p=1Pwp1-1s+nd+nf∑i=1swi+sip+yiop+∑i=1ndsipdziopd+∑i=1nfsipf+ziopfOther constraints of model (RIa) hold.Like the input-oriented model, the objective functionROa-1 consists of three terms. The first term minimizes the average efficiency of scenarios with weight importance of β. The second term minimizes the standard deviation of efficiencies with weight importance of 1- β. Finally, the third term calculates the total profit or loss resulted from scenarios.(1) Linear Robust Output-Oriented Model Step a ( L R O a ). By considering two positive variables Qs+ and Qs- and substituting it with the absolute function, the model is linearized. LROa-1min⁡β×Average+1-β×∑s=1SПs×Qs++Qs-+∑s=1SПs∑i=1nkisxsispλSsp+∑i=1nugiszsispuλssp+∑i=1nfviszsispf-λssp-∑i=1shisysispλssp-∑i=1ndbiszsispdλssp-∑i=1nfeiszsispf+λsspLROa-21P∑p=1Pwp1-1s+nd+nf∑i=1swi+sip+yiop+∑i=1ndsipdziopd+∑i=1nfsipf+ziopf-Average=Qs+-Qs-Other constraints of model (RIa) hold.


## 4.2. Step b: A DMU from Other DMUs Is under Investigation

In the previous section the under investigation DMU was one of the scenarios and we evaluated scenarios and selected the suitable one. In fact, model (RIa/ROa) calculates benefit or loss resulted from each scenario and we can select the best one accordingly. In this section, other DMUs are evaluated along with the selected scenario and therefore the number of DMUs increases by one (i.e.,m+1). Actually the best scenario is considered in model (RIb/ROb).


### 4.2.1. Robust Optimization for Input-Oriented DEA Model Step b (RIb)

The following model (RIb) evaluates other DMUs along with the strictly efficient DMU (i.e., the selected scenario as an ideal DMU) as a benchmark. Then the improvement methods can be presented for other DMUs based on their distance from the efficient frontier and image inefficient DMUs to the efficient frontier. The main difference of model (RIb) with model (RIa) is that in model (RIb) the number of DMUs is m+1. Another difference is that the objective function consists of two terms, i.e., the average efficiency and the standard deviation of efficiencies of the considered DMU in different periods respectively with importance weights β and 1-β. Note that the objective function of model (RIb) does not consider the cost since by taking the ideal DMU into consideration; we can rank DMUs and present improvement methods.RIb-1min⁡Φ0∗=β×Average+1-β×∑s=1SПs×1P⁡∑p=1Pwp1-1m+nu+nf∑i=1nwi-sip-xiop+∑i=1nusipuziopu+∑i=1nfsipf-ziopf-AverageRIb-2s.t.Average=∑s=1SПs×1P⁡∑p=1Pwp1-1m+nu+nf∑i=1nwi-sip-xiop+∑i=1nusipuziopu+∑i=1nfsipf-ziopfRIb-3xiop=∑j=1m+1xijpλjp+sip-i=1,…,n;p=1,…,P;j=1,…,m+1RIb-4yiop=∑j=1m+1yijpλjp-sip+i=1,…,s;p=1,…,P;j=1,…,m+1RIb-5ziopu=∑j=1m+1zijpuλjp+sipui=1,…,nu;p=1,…,P;j=1,…,m+1RIb-6zsiopf=∑j=1m+1zijpfλjp+sipf-i=1,…,nf;p=1,…,P;j=1,…,m+1RIb-7zsiopf=∑j=1mzijpfλjp-sipf+i=1,…,nf;p=1,…,P;j=1,…,m+1RIb-8zsiopd=∑j=1mzijpdλjp-sipdi=1,…,nd;p=1,…,P;j=1,…,m+1RIb-9sipu≥0,sip+≥0,sip-≥0,λjp≥0,sipf+≥0,sipf-≥0,sipd≥0(1) Linear Robust Input-Oriented Model Step b ( L R I b ). By considering two positive variables Qs+and Qs- and substituting it with the absolute function, the model is linearized. LRIb-1minΦ0∗=β×Average+1-β×∑s=1SПs×Qs++Qs-LRIb-2s.t.Average=∑s=1SПs×1P⁡∑p=1Pwp1-1m+nu+nf∑i=1nwi-sip-xiop+∑i=1nusipuziopu+∑i=1nfsipf-ziopfLRIb-31P⁡∑p=1Pwp1-1m+nu+nf∑i=1nwi-sip-xiop+∑i=1nusipuziopu+∑i=1nfsipf-ziopf-Average=Qs+-Qs-Other constraints of model (RIb) hold.


### 4.2.2. Robust Optimization for Output-Oriented DEA Model Step b (ROb)

R O b - 1 m i n  Φ 0 ∗ = β × A v e r a g e + 1 - β Φ 0 ∗ = × ∑ s = 1 S П s × 1 P ∑ p = 1 P w p 1 - 1 s + n d + n f ∑ i = 1 s w i + s i p + y i o p + ∑ i = 1 n d s i p d z i o p d + ∑ i = 1 n f s i p f + z i o p f - A v e r a g e R O b - 2 s . t .  A v e r a g e = ∑ s = 1 S П s × 1 P ∑ p = 1 P w p 1 - 1 s + n d + n f ∑ i = 1 s w i + s i p + y i o p + ∑ i = 1 n d s i p d z i o p d + ∑ i = 1 n f s i p f + z i o p fOther constraints of model (RIb) hold.(1) Linear Robust Output-Oriented Model Step b ( L R O b ). By considering two positive variables Qs+ and Qs- and substituting them with the absolute function, the model is linearized. LROb-1minΦ0∗=β×Average+1-β×∑s=1SПs×Qs++Qs-LROb-2s.t.Average=∑s=1SПs×1P∑p=1Pwp1-1s+nd+nf∑i=1swi+sip+yiop+∑i=1ndsipdziopd+∑i=1nfsipf+ziopfLROb-31P∑p=1Pwp1-1s+nd+nf∑i=1swi+sip+yiop+∑i=1ndsipdziopd+∑i=1nfsipf+ziopf-Average=Qs+-Qs-Other constraints of model (RIb) hold.


## 4.2.1. Robust Optimization for Input-Oriented DEA Model Step b (RIb)

The following model (RIb) evaluates other DMUs along with the strictly efficient DMU (i.e., the selected scenario as an ideal DMU) as a benchmark. Then the improvement methods can be presented for other DMUs based on their distance from the efficient frontier and image inefficient DMUs to the efficient frontier. The main difference of model (RIb) with model (RIa) is that in model (RIb) the number of DMUs is m+1. Another difference is that the objective function consists of two terms, i.e., the average efficiency and the standard deviation of efficiencies of the considered DMU in different periods respectively with importance weights β and 1-β. Note that the objective function of model (RIb) does not consider the cost since by taking the ideal DMU into consideration; we can rank DMUs and present improvement methods.RIb-1min⁡Φ0∗=β×Average+1-β×∑s=1SПs×1P⁡∑p=1Pwp1-1m+nu+nf∑i=1nwi-sip-xiop+∑i=1nusipuziopu+∑i=1nfsipf-ziopf-AverageRIb-2s.t.Average=∑s=1SПs×1P⁡∑p=1Pwp1-1m+nu+nf∑i=1nwi-sip-xiop+∑i=1nusipuziopu+∑i=1nfsipf-ziopfRIb-3xiop=∑j=1m+1xijpλjp+sip-i=1,…,n;p=1,…,P;j=1,…,m+1RIb-4yiop=∑j=1m+1yijpλjp-sip+i=1,…,s;p=1,…,P;j=1,…,m+1RIb-5ziopu=∑j=1m+1zijpuλjp+sipui=1,…,nu;p=1,…,P;j=1,…,m+1RIb-6zsiopf=∑j=1m+1zijpfλjp+sipf-i=1,…,nf;p=1,…,P;j=1,…,m+1RIb-7zsiopf=∑j=1mzijpfλjp-sipf+i=1,…,nf;p=1,…,P;j=1,…,m+1RIb-8zsiopd=∑j=1mzijpdλjp-sipdi=1,…,nd;p=1,…,P;j=1,…,m+1RIb-9sipu≥0,sip+≥0,sip-≥0,λjp≥0,sipf+≥0,sipf-≥0,sipd≥0(1) Linear Robust Input-Oriented Model Step b ( L R I b ). By considering two positive variables Qs+and Qs- and substituting it with the absolute function, the model is linearized. LRIb-1minΦ0∗=β×Average+1-β×∑s=1SПs×Qs++Qs-LRIb-2s.t.Average=∑s=1SПs×1P⁡∑p=1Pwp1-1m+nu+nf∑i=1nwi-sip-xiop+∑i=1nusipuziopu+∑i=1nfsipf-ziopfLRIb-31P⁡∑p=1Pwp1-1m+nu+nf∑i=1nwi-sip-xiop+∑i=1nusipuziopu+∑i=1nfsipf-ziopf-Average=Qs+-Qs-Other constraints of model (RIb) hold.


## 4.2.2. Robust Optimization for Output-Oriented DEA Model Step b (ROb)

R O b - 1 m i n  Φ 0 ∗ = β × A v e r a g e + 1 - β Φ 0 ∗ = × ∑ s = 1 S П s × 1 P ∑ p = 1 P w p 1 - 1 s + n d + n f ∑ i = 1 s w i + s i p + y i o p + ∑ i = 1 n d s i p d z i o p d + ∑ i = 1 n f s i p f + z i o p f - A v e r a g e R O b - 2 s . t .  A v e r a g e = ∑ s = 1 S П s × 1 P ∑ p = 1 P w p 1 - 1 s + n d + n f ∑ i = 1 s w i + s i p + y i o p + ∑ i = 1 n d s i p d z i o p d + ∑ i = 1 n f s i p f + z i o p fOther constraints of model (RIb) hold.(1) Linear Robust Output-Oriented Model Step b ( L R O b ). By considering two positive variables Qs+ and Qs- and substituting them with the absolute function, the model is linearized. LROb-1minΦ0∗=β×Average+1-β×∑s=1SПs×Qs++Qs-LROb-2s.t.Average=∑s=1SПs×1P∑p=1Pwp1-1s+nd+nf∑i=1swi+sip+yiop+∑i=1ndsipdziopd+∑i=1nfsipf+ziopfLROb-31P∑p=1Pwp1-1s+nd+nf∑i=1swi+sip+yiop+∑i=1ndsipdziopd+∑i=1nfsipf+ziopf-Average=Qs+-Qs-Other constraints of model (RIb) hold.


## 5. Case Study Implementation and Performance Evaluation

In this section, the performance of the proposed robust dynamic DEA approach is investigated via a case study taken from NANIWA (http://www.naniwa.ir) appliances production plant. The mentioned firm aims at evaluating its 35 suppliers in 4 time periods based on environmental, social, and economic criteria. For the purpose of evaluation, for each supplier as a DMU, 4 inputs, 3 desirable and undesirable relationships, and 4 outputs are considered.Inputs (1)
Price offered by suppliers (as an economic criterion): It is the money (in $1000) that is paid to suppliers for each unit of products. (2)
Cost of recoverable packages (as an environmental criterion): This input is the cost (in $100) that is imposed to the company by the supplier for using recoverable packages. It is worth noting that it is mandatory for Naniwa Company to ship their products in suitable pallets with recoverable packages to prevent damaging the environment. (3)
Final transportation cost (as an economic criterion): This is the final cost (in $100) which is imposed by the supplier to the firm for transportation of shipped pallets. The farthest the supplier is and the less accessible the paths are, the more cost imposed. This cost is the main concern of the decision-makers in the company. (4)
Work safety and labor health (as a social criterion): This is the cost that each supplier pays for dangers that exist and accidents which happen in the workplace. The less damage and casualties are, the less cost is paid by each supplier.Relationships (1)
Used technology in the production line of suppliers (as a desirable relationship and an economic criterion): The used technology is scored by expert’s opinion using 9-point Likert spectrum according to Table1. Likert scale is used to convert qualitative factors into quantitative values [42]. There are variety of scales which can be rated as 1 to 5, 1 to 7, and 1 to 9. Valuation of factors in this scale is performed according to concept of each factor [43]. (2)
Green research and development (as a free relationship and an environmental criterion): The corresponding budget per year (in $100). Green research and development is a dual-role factor which plays the role of both undesirable and desirable factors. Green research and development can be considered as an undesirable criterion since it is cost of performing green researches. On the other hand, green research and development is a desirable criterion, because it implies innovations in manufacturing green products and services and environmental efficiency enhancement. (3)
Shortages (as an undesirable relationship and an economic criterion): The amount of shipments (in terms of pallets) that have not delivered in the past period and should be met in the next period by the supplier.Table 1
9-point Likert spectrum for Technologic power.
Value 9 7 5 3 1 2-4-6-8 Technology High Technology Good Technology Medium Technology Weak Technology Very weak Technology Intermediate values for TechnologyOutputs (1)
Obtaining ISO certificates and observance of standards: this kind of output is scored by expert’s opinion using 9-point spectrum with respect to qualitative and environmental criteria and standards and also workplace standards. Table2 shows the 9-point spectrum. (2)
Quality (as an economic criterion): Quality of products is evaluated by Likert scale. In Table3, using a 9-point Likert scale, valuation of quality of parts supplied by suppliers is presented. (3)
Supply capacity (g_2): maximum amount of materials that supplier can send to Naniwa Co. (4)
Efficiency of energy consumption (as an environmental criterion): Efficiency of the energy consumption is the third output which is an environmental criterion. To determine an appropriate scale for evaluating efficiency of energy consumption we apply a scoring method which is shown by A+++ to G scale. The letter A+++ shows the lowest energy consumption. The letter G indicates the highest energy consumption. Using 100-point scale. Table4 shows the energy consumption of the suppliers.Table 2
9-point Likert spectrum for Standards.
Value 9 7 5 3 1 2-4-6-8 ISO standards High Standards Good Standards Medium Standards Weak Standards Very Weak Standards Intermediate values for StandardsTable 3
9-point Likert spectrum for Quality.
Value 9 7 5 3 1 2-4-6-8 Quality High Quality Good Quality Medium Quality Weak Quality Very Weak Quality Intermediate values for QualityTable 4
Spectrum for efficiency of the energy consumption.
Score 100 90 80 70 60 50 40 30 20 10 Efficiency of the energy consumption A+++ A++ A+ A B C D E F GFigure4 illustrates the proposed dynamic DEA approach on the case study to evaluate suppliers in periods 2011 to 2014. The inputs, outputs, and relationships between periods is illustrated in this figure.Figure 4
Demonstration of the proposed dynamic DEA on the considered case study.The inputs, relationships, and outputs data for 35 suppliers in 4 time periods are presented in Table5 in the Appendix.Table 5
The inputs, relationships and outputs data for 35 suppliers in 4 time periods.
DMU 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 2011 IN1 26 31 29 30 26 34 32 16 29 30 33 28 24 31 29 26 25 29 28 31 35 15 33 21 26 24 28 23 30 27 26 31 25 29 33 IN2 15 18 23 22 20 19 14 9 17 18 19 25 22 20 18 17 16 23 19 22 19 10 19 13 15 16 17 19 20 18 12 16 16 18 21 IN3 13 12 11 10 9 6 7 2 8 10 6 9 9 11 14 5 8 6 9 5 7 3 9 8 12 5 6 7 8 9 10 5 8 12 9 IN4 10 9 7 8 9 6 4 8 10 9 8 11 8 9 9 10 6 9 7 8 10 2 9 9 8 6 6 7 8 9 10 6 8 8 9 OUT1 4 4 6 5 8 4 9 6 5 7 6 5 4 4 5 6 7 8 6 3 4 9 5 8 7 4 5 7 6 5 4 6 5 9 8 OUT2 8 6 5 9 5 6 9 7 8 7 5 7 8 6 7 9 6 8 7 7 9 9 7 7 6 8 7 8 7 9 9 7 6 8 7 OUT3 78 53 59 64 69 71 88 68 63 59 55 49 69 72 66 67 81 71 63 65 73 95 62 56 84 79 83 78 66 69 59 74 95 76 85 OUT4 60 90 80 80 90 90 60 100 80 90 60 70 90 80 80 60 70 50 60 50 40 100 70 70 80 60 60 50 70 60 40 70 60 50 80 LINK1 8 7 8 6 7 9 6 9 5 5 7 9 7 8 6 8 9 5 5 6 7 9 9 8 6 5 8 8 6 7 8 6 6 7 8 LINK2 28 31 30 32 29 28 35 18 26 26 27 29 30 33 27 25 29 30 27 29 27 15 25 26 37 32 34 26 29 29 28 26 33 34 25 LINK3 20 16 17 18 13 15 15 2 15 18 16 18 15 13 6 8 9 7 6 8 11 3 12 19 15 13 14 6 16 9 11 12 14 15 10  2012 IN1 19 33 32 27 29 29 28 30 32 33 36 29 28 32 31 29 28 32 33 28 30 15 29 33 28 30 29 29 27 32 30 33 29 25 38 IN2 12 16 25 24 22 20 16 18 16 17 20 26 19 23 22 19 18 20 16 18 20 10 16 21 19 14 20 16 15 20 16 20 18 16 18 IN3 3 9 13 12 10 5 6 7 7 9 5 7 8 14 19 7 9 5 7 9 10 5 7 9 14 9 5 7 9 7 9 7 10 9 10 IN4 4 11 8 9 8 9 10 6 9 7 9 10 9 13 12 9 8 10 8 6 9 5 11 10 9 8 7 5 6 7 9 8 6 7 10 OUT1 9 6 7 8 6 6 5 7 6 8 6 7 8 6 5 8 6 7 5 8 6 9 7 8 5 6 6 8 7 7 6 5 8 6 7 OUT2 9 6 7 8 6 5 5 8 9 9 6 9 8 7 7 6 8 6 9 8 7 9 6 6 7 8 5 8 6 7 8 9 7 8 8 OUT3 95 59 60 70 77 66 68 52 70 59 63 52 73 78 72 69 73 69 74 72 69 89 73 66 78 66 91 65 73 72 60 85 80 66 80 OUT4 100 80 80 80 90 80 50 60 80 70 50 70 70 80 80 70 70 50 60 60 40 100 70 70 70 60 60 60 80 60 50 70 70 50 80 LINK1 9 6 7 8 8 8 7 7 6 6 7 8 6 7 7 7 8 6 6 7 8 9 8 7 7 7 7 7 7 6 8 7 7 8 7 LINK2 12 30 28 31 33 29 29 25 19 28 34 36 33 28 34 29 25 32 33 30 26 20 30 31 28 26 29 30 32 25 27 29 32 26 28 LINK3 5 12 10 11 17 16 15 13 17 16 14 11 10 11 15 12 12 17 14 12 16 5 10 10 17 10 12 11 18 14 13 15 18 12 10  2013 IN1 31 32 34 26 30 26 29 32 35 32 30 26 30 30 29 22 29 31 30 25 32 26 28 32 29 31 30 28 26 33 28 30 26 24 15 IN2 18 18 27 25 24 22 19 19 19 18 22 28 17 25 25 17 19 24 19 19 22 19 18 23 17 18 21 17 17 22 18 21 19 15 9 IN3 14 10 14 13 11 8 9 8 8 10 8 9 9 11 15 8 10 7 8 8 12 10 8 10 12 8 8 9 10 6 7 8 8 4 2 IN4 15 14 12 11 10 10 12 8 7 9 10 12 10 15 14 11 10 14 9 9 10 10 15 14 10 11 12 10 5 8 8 11 9 10 4 OUT1 7 9 7 9 8 9 9 8 7 7 9 8 9 6 9 7 8 8 9 9 8 8 6 9 8 7 6 9 9 7 8 8 6 7 9 OUT2 6 8 9 8 6 9 7 8 6 8 7 9 9 8 6 7 9 9 8 7 9 8 6 7 8 9 9 8 8 9 8 7 9 8 9 OUT3 63 70 72 64 86 70 64 65 65 60 70 60 65 85 80 75 80 70 65 83 82 70 65 80 81 75 76 55 65 80 50 80 77 65 95 OUT4 70 70 80 60 90 70 50 60 70 50 40 70 60 80 70 70 70 70 60 60 50 60 70 70 70 80 60 50 80 60 60 70 70 60 100 LINK1 8 7 7 8 9 8 7 9 6 5 7 9 6 7 8 5 8 5 8 7 8 6 8 7 8 6 8 7 8 6 8 8 7 8 9 LINK2 33 32 27 29 30 34 31 39 28 30 30 36 33 29 31 30 26 27 29 30 31 36 34 33 29 30 32 29 37 29 34 33 26 28 10 LINK3 15 19 21 20 19 18 17 16 14 15 19 18 16 17 15 12 18 17 13 15 16 14 10 12 14 11 13 15 16 14 12 15 16 12 2  2014 IN1 20 34 37 29 34 33 32 36 37 34 34 35 34 32 33 28 32 34 33 31 30 30 32 34 30 33 32 30 30 35 33 32 28 28 15 IN2 14 22 30 28 27 26 25 20 23 24 20 26 20 27 29 20 22 26 20 21 25 21 19 26 19 21 23 21 20 24 21 22 20 17 12 IN3 4 14 15 15 13 12 12 10 10 12 10 10 11 14 17 16 12 18 10 11 15 12 9 14 15 9 7 10 12 14 10 16 9 10 5 IN4 8 15 14 13 12 12 9 10 12 10 10 11 10 18 13 12 11 13 10 10 13 12 14 11 12 11 14 12 9 11 10 14 11 9 7 OUT1 9 7 7 9 9 8 8 8 9 7 7 8 9 9 7 8 6 9 7 9 8 8 9 8 7 7 8 9 9 8 8 7 9 8 9 OUT2 9 9 9 9 7 9 7 9 7 9 8 9 9 8 8 7 9 9 8 8 9 8 8 8 9 9 9 9 9 9 8 9 9 8 9 OUT3 89 59 63 72 73 72 68 68 60 68 67 73 72 66 64 69 65 43 70 72 71 74 73 70 72 66 67 59 68 75 63 78 69 59 95 OUT4 100 90 80 80 90 70 60 60 90 70 50 60 60 90 70 80 70 80 70 60 50 50 60 80 70 80 60 70 100 60 60 80 70 70 100 LINK1 9 7 7 8 9 9 8 9 7 7 8 9 7 8 8 6 8 6 9 7 9 7 8 6 9 7 8 8 8 8 8 8 9 8 9 LINK2 20 30 29 30 32 32 33 35 29 32 32 35 34 30 33 32 28 29 30 29 30 35 36 34 31 32 34 32 30 32 36 30 25 31 22 LINK3 5 9 11 10 14 16 13 11 12 10 14 16 16 18 5 17 16 18 15 10 10 13 12 14 16 7 10 21 18 14 11 13 15 10 3In Table6, the DMUs (i.e., 35 suppliers) are evaluated by making use of the existing input-oriented dynamic DEA model on data shown in the Table 5 presented in the Appendix. The DEA model is selected according to the DM’s approach for which he/she want to present their proposed improvement plans. Table 6 shows the efficiency values for 35 suppliers in 4 periods from 2011 to 2014.Table 6
Efficiency values for 35 suppliers in 4 periods (Input-oriented dynamic DEA model).
DMU ϕ 2011 ϕ 2012 ϕ 2013 ϕ 2014 Φ j RANK 1 0.856 1 0.986 1 0.9605 1 2 0.763 0.943 0.915 0.966 0.8967 4 3 0.88 0.849 0.965 0.824 0.8795 11 4 0.867 0.825 0.814 0.894 0.85 26 5 0.728 0.831 0.824 0.872 0.8137 33 6 0.809 0.941 0.769 0.881 0.85 26 7 0.911 0.817 0.964 0.764 0.864 19 8 1 0.866 0.813 0.821 0.875 13 9 0.766 0.835 0.849 0.938 0.847 29 10 0.849 0.961 0.827 0.933 0.846 31 11 0.867 0.985 0.863 0.74 0.8637 20 12 0.857 0.938 0.917 0.768 0.87 16 13 0.892 0.846 0.965 0.884 0.8967 4 14 0.893 0.851 0.824 0.825 0.8482 28 15 0.846 0.837 0.764 0.864 0.8277 32 16 0.937 0.84 0.819 0.897 0.8732 15 17 0.967 0.869 0.893 0.831 0.89 7 18 0.955 0.815 0.84 0.843 0.8632 21 19 0.934 0.834 0.933 0.817 0.8795 11 20 0.871 0.769 0.968 0.934 0.8855 9 21 0.768 0.941 0.824 0.92 0.8632 22 22 1 1 0.866 0.819 0.9212 3 23 0.869 0.911 0.814 0.8349 0.8572 24 24 0.942 0.771 0.867 0.822 0.8505 25 25 0.852 0.764 0.789 0.796 0.8002 34 26 0.789 0.859 0.984 0.753 0.8462 30 27 0.844 0.915 0.943 0.749 0.8627 23 28 0.809 0.864 0.972 0.921 0.8915 6 29 0.78 0.91 0.928 0.966 0.8842 10 30 0.922 0.752 0.846 0.946 0.8665 18 31 0.818 0.92 0.837 0.925 0.875 13 32 0.846 0.881 0.819 0.933 0.8697 17 33 0.856 0.825 0.935 0.928 0.886 8 34 0.736 0.714 0.966 0.784 0.8 35 35 0.855 0.891 1 1 0.9365 2The results of Table6 show that none of the suppliers have unit efficiency values in all 4 periods. Therefore, none of them are strictly efficient. To construct a strictly efficient unit as a benchmark for other units, in each period the DMU with unit efficiency is selected as a candidate. As a result, 8 scenarios are generated for ideal DMU which are different combinations of DMUs (suppliers) with unit efficiency in each period. These resulting scenarios are presented in Table 7 with probability of 0.125 for each one. If these scenarios are taken into consideration and evaluated along with other 35 suppliers, the efficiency of scenarios becomes one because they consist of DMUs with unit efficiencies in all periods. Therefore, we can claim that the existing dynamic DEA model cannot evaluate and rank the scenarios. In the proposed model (RIa), since all scenarios have unit efficiency, the average efficiency is also one and the deviation of efficiencies is zero. Therefore, we set β=1,1-β=0. To deal with this difficulty, a punishment (Cost) and encouragement (Income) value is considered for the inputs, outputs, and relationships of each scenario.Table 7
Different scenarios for Ideal DMU.
Ideal DMU Period 2011 Period 2012 Period 2013 Period 2014 Scenario 1 Supplier 22 Supplier 22 Supplier 35 Supplier 1 Scenario 2 Supplier 22 Supplier 22 Supplier 35 Supplier 35 Scenario 3 Supplier 22 Supplier 1 Supplier 35 Supplier 1 Scenario 4 Supplier 22 Supplier 1 Supplier 35 Supplier 8 Scenario 5 Supplier 8 Supplier 22 Supplier 35 Supplier 1 Scenario 6 Supplier 8 Supplier 22 Supplier 35 Supplier 35 Scenario 7 Supplier 8 Supplier 1 Supplier 35 Supplier 1 Scenario 8 Supplier 8 Supplier 1 Supplier 35 Supplier 35The costs and incomes associated with inputs, outputs, and relationships are presented in Table8 with the following notations:Table 8
Costs and incomes associated with inputs, outputs and relationships.
Parameters k 1 A L L k 2 A L L k 3 A L L k 4 A L L g 1 A L L v 1 A L L Costs 10 5 7 3 11 9  Parameters h 1 A L L h 2 A L L h 3 A L L h 4 A L L e 1 A L L b 1 A L L  Incomes 20 15 18 12 15 18k j A L L: Unit cost for input j for all scenarios (j: price, packaging, transportation cost, and workforce health cost): this is a penalty that is imposed to a scenario by the decision-maker for each unit of inputs.h j A L L: Unit income for output j for all scenarios (j: quality, production capacity, the number of acquired ISO and standards, and efficiency of the energy consumption): this is the encouragement that is considered for a scenario for each unit of outputs.g 1 A L L: Unit cost of shortages in shipped pallets for all scenarios (undesired relationships): this is the penalty that is decided by the decision-maker for each unit of this undesired relationship. In this paper for backlogs or goods shipped with a delay a penalty is considered for the supplier in that period.v 1 A L L: Unit cost of green research and development for all scenarios (free relationships): if the free relationship be an undesired relationship a penalty value will be assigned for each unit of this relationship. In our case study, if the green R&D is considered as an undesired relationship for the under investigation DMU, a penalty is considered for that supplier for each unit of this relationship.e 1 A L L: Unit income of green research and development for all scenarios (free relationships): if the free relationship is a desired relationship an encouragement value will be assigned for each unit of this relationship. In our case study, if the green R&D is considered as a desired relationship for the under investigation DMU, an encouragement value is considered for that supplier for each unit of this relationship.b 1 A L L: Unit income of technological power for all scenarios (desirable relationships).As mentioned earlier, the existing dynamic DEA model cannot rank different scenarios for the ideal supplier. In this paper, first we evaluate and rank these scenarios through the proposed input-oriented robust dynamic DEA model(RIa). In Table 9, the punishment and encouragement for each scenario resulting from model RIa is presented. As presented in Table 9, the second scenario is the best scenario since it has the maximum value of benefit. Generally, if the objective function of model (RIa) is positive, the scenario will generate loss while it is negative the scenario will make benefits. If all scenarios generate loss, the scenario with minimum loss will be selected. The values of landa obtained from model (RIa) admit that the scenarios can be considered as a benchmark. In the next step, the second scenario which is the best scenario is considered as the ideal DMU and it is evaluated along with other DMUs (suppliers) through model (RIb). The ideal DMU which is actually the ideal scenario is considered as a unique benchmark which owns both the property of a real supplier in that it consists of some periods and the property of a virtual supplier in that it is strictly efficient and consists of periods with unit efficiency. Therefore, the proposed ideal DMU can present improvement plans for all suppliers and can rank the suppliers with same efficiency. The results obtained from model (RIb) are reported in Table 10. As presented in Table 10, the ideal DMU gives the ability of ranking the suppliers that had the same rank in Table 6. From Table 6 we can see that suppliers 2 and 13, 19 and 3, 8 and 31, 6 and 4, two by two have the same rank because the total efficiency of these DMUs are equal. Using the proposed model (RIb) for evaluating the suppliers and constructing an ideal DMU help us to rank the suppliers. This ranking considers the standard deviation of efficiencies in calculating the efficiency of each supplier. Furthermore, the proposed ideal DMU gives us the opportunity of presenting improvement methods for all DMUs. As a matter of fact, the improvement methods are presented based on their distance from the ideal DMU (supplier). The high ranked scenario resulting from model (RIa), which is the second scenario, has maximum distance from other suppliers and it is the worst case that can be happen for an ideal scenario based on which improvement plans can be presented. Through model (RIb) other suppliers can be ranked based on their distance from this worst case ideal scenario. Therefore, we could claim that the resulting ranks and improvement plans cannot get worse when each of the other scenarios, which could be happen with a given probability, is considered thus resulting in a robust solution.Table 9
Evaluating different scenarios for the Ideal DMU (ModelRIa).
Scenario 1 2 3 4 5 6 7 8 Worst case Efficiency 1 1 1 1 1 1 1 1 1 Benefit/loss 10027 10988 10103 9655 10320 10768 9435 9883 10147.375Table 10
Evaluating suppliers along with the ideal supplier (ModelRIb).
DMU 1 2 3 4 5 6 7 8 9 Efficiency 0.489 0.478 0.445 0.354 0.317 0.359 0.412 0.437 0.341  RANK 2 5 13 28 34 27 20 15 30  DMU 10 11 12 13 14 15 16 17 18  Efficiency 0.325 0.406 0.428 0.475 0.348 0.321 0.432 0.469 0.401  RANK 32 21 17 6 29 33 16 8 22  DMU 19 20 21 22 23 24 25 26 27  Efficiency 0.449 0.462 0.392 0.481 0.377 0.362 0.309 0.332 0.384  RANK 12 10 23 4 25 26 35 31 24  DMU 28 29 30 31 32 33 34 35 Ideal DMU  Efficiency 0.472 0.458 0.416 0.439 0.422 0.466 0.305 0.484 0.5  RANK 7 11 19 14 18 9 36 3 1


## 6. Conclusions and Future Research Directions

This paper proposed a new DEA approach for evaluating suppliers based on sustainable supplier criteria such as social, economic, and environmental criteria. The proposed model considers suppliers in different periods thus leading to a dynamic DEA model. Existing dynamic DEA models just present improvement plans and do not have the ability of ranking DMUs. The proposed model apart from having the ability of ranking DMUs can present improvement plans. In most DEA models, when different time periods are considered in evaluating DMUs, no DMUs can be introduced as a strictly efficient unit which is efficient in all periods. Our contribution is introducing a new method for constructing the ideal DMU such that apart from the previous definition for ideal DMU which considers one of the DMUs which has unit efficiency in all periods as an ideal DMU, a virtual DMU is considered as an ideal DMU. The new proposed ideal DMU has the ability of presenting an improvement method for all suppliers and also ranking the suppliers with the same efficiency. The proposed ideal DMU introduces a strictly efficient unit by building a combination of DMUs with unit efficiency in each period. It is possible that multiple DMUs have unit efficiency in a period. Therefore, different combinations of these units lead to different scenarios for the ideal DMU each of which can happen with a specific probability. The existing dynamic DEA model cannot evaluate and rank the scenarios which all have unit efficiency. To deal with these scenarios, a scenario-based robust optimization model for the dynamic DEA is developed which is capable of ranking the scenarios based on a punishment and encouragement value assigned to each scenario.The proposed robust method is implemented in two steps and it is is able to obtain a solution which is immunized against different scenarios exist in constructing the ideal DMU. In the first step, at each time one of the scenarios is under investigation. The high ranked scenario which has more distance from other DMUs is selected as a unique benchmark based on which the improvement plan is presented in the worst case. In fact, in the second step other DMUs along with the worst case scenario are under investigation and improvement plans are proposed based on their distance from the ideal DMU. Therefore, the proposed method can rank the suppliers with the same efficiency and propose an improvement plan which is robust against different scenarios which could be considered for the ideal DMU.For future study on this work, apart from the uncertainty in different scenarios for ideal DMU, one can consider the input, output, and relationship values as uncertain parameters which can vary in a convex uncertainty set. Then a hybrid robust optimization approach, i.e., a hybridization of the scenario-based and robust counterpart methods, can be developed to deal with these uncertainties.



---

*Source: 7625025-2018-12-09.xml*