There are free lunches

1 INTRODUCTION . No-Free-Lunch ( NFL ) Theorems Wolpert & Macready ( 1997 ) Wolpert ( 2002 ) state that “ all optimization algorithms perform equally well when their performance is averaged across all possible problems ” . Some researchers Alabert et al . ( 2015 ) argued that the necessary conditions for NFL are too restrictive . In practice , there must be some algorithms that perform better than others . However , no one doubt that no algorithm can optimally solve all tasks . The implications of the NFL Theorems are very disappointing , as an universal algorithm is a dream in the field of Artificial Intelligence ( AI ) . This paper proposes the “ There are free lunches ” ( TAFL ) Theorem . The TAFL Theorem states that some algorithms are optimal for all tasks , in the condition that tasks are given in a specific order . We also conclude that the difficulty of solving a new task can decrease when the number of solved tasks increases . The TAFL Theorem implies a very different way of solving AI problems compared to the NFL Theorems . It is not a matter that algorithms at our hands are not powerful or suitable enough for a specific task . One can use an algorithm of limited intelligence to solve a complex problems step by step . We introduce the necessary notation in Section 2 and provide the TAFL Theorem in Section 3 . We analyse the task order mentioned in the TAFL Theorem and point out we can reduce calls of learning algorithms in Section 4 . An application of the TAFL Theorem is provided in Section 5 . Finally , we discuss the implications of the TAFL Theorem in Section 6 and conclude the paper in 7 . 2 PRELIMINARIES . We restrict attention to supervised learning . But the discussion is also applicable for optimization and search fields . The entire sample space is denoted as D. The distribution space is denoted as F . A distribution from F is a mapping f : x → y . A sampling strategy S samples a set of training samples from f , denoted as d = { ( x0 , y0 ) , ... , ( xm , ym ) } , d ∈ D. We consider the sampling strategy is unique for all tasks . A supervised learning algorithm generates a mapping ( or called a hypothesis ) h : x→ y to approximate the distribution f , in terms of off-training-set error . The training samples d are provided to an algorithm La to generate a set of h. The possibility of the algorithm La generating a specific h on d is denoted as P ( h|d , La ) . No-Free-Lunch Theorems state the total off-training-set error across all possible f is always the same for arbitrary algorithms La and Lb.∑ f Eote ( La|d , f ) = ∑ f Eote ( Lb|d , f ) ( 1 ) The off-training-set error Eote ( La|d , f ) is given in Eq.2 , where I ( · ) is 1 when · is ture , otherwise 0 . Eote ( La|d , f ) = ∑ h ∑ di∈f , di /∈d P ( di ) I ( h ( di ) 6= f ( di ) ) P ( h|d , La ) ( 2 ) 3 THE TAFL THEOREM . We consider that a distribution can be parametrized as f ( : θ ) . For a specific algorithm La , there is a set of distribution { fa ( : θa ) |θa ∈ R } which can be optimally solved by La . The TAFL Theorem is described as below . La and Lb are the optimal algorithms , in terms of off-training-set error , for distribution sets { fa ( : θa ) |θa ∈ R } and { fb ( : θb ) |θb ∈ R } respectively . fa ( : θa ) has the form of y = ga ( ∑ i x iθia + θ b a ) . ga can be any continuous nonconstant function . Given a specific fb ( : θ∗b ) and its samples d ∗ b , we can find a serial of tasks , representing by a serial of sets of samples { d0a , ... , dna } . La is the optimal algorithm for each of dia and if we solve each dia in the provided order , then Eote ( La|dna , fb ( : θ∗b ) ) = Eote ( Lb|d∗b , fb ( : θ∗b ) ) . The proof is given as below . According to the universal approximation theorem Hornik et al . ( 1989 ) , fb ( : θ∗b ) can be arbitrarily accurate approximated by a continuous nonconstant function . As fa ( x : θa ) has the form y = ga ( ∑ i x iθia + θ b a ) , the approximation has the form : y0 = x y1 = { yi1|yi1 = fa ( y0 : θ1 , ia ) , i = 0 , ... , m } . . . yn = { yin|yin = fa ( yn−1 : θn , ia ) , i = 0 , ... , k } fb ( x : θ ∗ b ) = yn ( 3 ) Each mapping f : ym−1 → ym has the form of fa ( x : θa ) , therefore they can be optimally solved by La . But for each f : fm−τ → fm , τ ≥ 2 , there is no guarantee that they have the form of fa ( x : θa ) , therefore they may not be optimally solved by La . As each f : ym−1 → ym has the form of fa ( x : θa ) , they are also tasks generated by the distribution fa ( : θa ) . Hence , if a serial of tasks is provided in a order { f : ym−1 → ym|m = 1 , ... , n } , La can solve fb ( : θ∗b ) just as good as Lb . The above proof does not conflict with the universal approximation theorem . The universal approximation theorem is not about the off-training-set error . Therefore , there is no guarantee that a feedforward network is optimal in terms of off-training-set error for any function . One may misunderstand the serial of tasks is similar to hidden layers of a neural network . They are totally different concepts . Each task in the serial is sampled from the task distribution space . Hence , they can be considered as real applications . For example , one task is a OCR task , and another task is a translation task . But the hidden layers of a neural network are not related to real applications , only the outputs of the whole network are related to real applications . 4 ANALYSIS OF THE TASK ORDER . Next , we analyse the task order mentioned in the TAFL Theorem . We first describe our conclusion as below . La is the optimal algorithm in terms of off-training-set error for a set of distribution { fa ( : θa ) |θa ∈ R } . fa ( : θa ) has the form of y = ga ( ∑ i x iθia + θ b a ) . Given a task of distribution fb ( : θ∗b ) , TO ∗ = { d0∗a , ... , dn∗a } is the correct task order for algorithm La to optimally solve fb ( : θ∗b ) . The task order TO ∗ is a sample among all possible task order samples . For a new given task , we sample a set of task orders TOi randomly or following some distribution until the TO∗ is sampled . For each sampled TOi , we call La to solve each of dia in it . We declare that the number of calls of La to solve each sampled dia until d n∗ a is solved can decrease when the number of previously solved tasks increases . Roughly speaking , we can learn how to combine models generated in previous tasks to solve new tasks to reduce the calls of L. The detailed reasons are listed below . 1 We can use a set M to store models of all solved di . Therefore , if a new task needs to solve a di which model is already in M , we can skip running La on it . 2 We can construct a new kind of task based on M . This kind of task , denoted as abstract tasks , learns a mapping from di to a sub-set of models in M : fA : di → m , m ⊂ M , where m should contain models of the correct task order for solving di . The loss of this mapping is shown in Eq.4 . L ( fA|La , fa , M ) = ∑ di ∑ m Eote ( La|di , fa ( m ( x ) : θa ) ) P ( m|di , fA ) ( 4 ) For a given d∗b , generated by fb ( : θ ∗ b ) , if the model of a mapping fA can recognize it and give a set of models m∗ which guarantees that fb ( : θ∗b ) = fa ( m ∗ ( x ) : θa ) , then La can optimally solve this task directly . According to the TAFL Theorem , we know there must be an algorithm LA which is capable to optimally solve all possible Fabs = { fA , fB , ... } . Each training sample of fA is a task in the distribution space F , and the model of fA is optimized to minimize off-training-set error . Therefore , the model of a single fA can be used to generate m for a set of tasks in F . It means after learning a single task fA , we can decrease the number of running La on a set of tasks in F . Following this idea , we can also construct fA2 : ( di , m ) → mA2 , mA2 ⊂ MA2 , where MA2 stores models generated in solving tasks in the distribution space Fabs . All possible fA2 form a new distribution space Fabs2 . The model of a single fA2 can be used to generate mA2 for a set of tasks fA ∈ Fabs which decreases the number of running algorithm LA on those tasks . Generally speaking , for the model of a task in distribution Fabsi , it can be used to decrease the number of running algorithms LAi−1 on a set of tasks in Fabsi−1 . Please note , according to the TAFL Theorem , we can use the same algorithm for all LAi . 3 We consider situations that fAi ∈ Fabsj or fa ∈ Fabsj for some i , a and j . It means models of a distribution space can be used in another distribution space . We consider two situations . In the first situation , fAi , from the distribution space Fabsi , is a part of a task order TOA j . TOA j is sampled for solving a task fAj that comes from the distribution space Fabsj . In this situation , the model of fAi can be used directly to avoid running an algorithm for solving fAi again . In the second situation , fAi is equal to fAj . Therefore , we can avoid learning fAj and use the model of fAi to solve tasks that fAj are required to solve . In those two situations , the calls of learning algorithms are decreased . The elements m of inputs and outputs of fAi and fAj are from different model sets . One may argue that how fAi ∈ Fabsj is possible . Here we suggest some simple ways . For example , we can put all models in the same set , or we can use an encoding function to map any m to a real-valued embeddingE : m→ r. An example is given in the section.5 which shows a way to allow fAi ∈ Fabsj .

Summary: The paper provides a new optimization theorem called "There Are Free Lunches" which in particular argues the NFL theorem. Based on the assertion, although the NFL states that averaged overall data distribution all algorithms have the same error rate, the TAFL theorem proposes that some algorithms can achieve the best performance if tasks are given in special order. Also, it is declared that if more problems are solved the difficulty of the new task will be decreased. They also mentioned an example to use the theorem. The proof of the TAFL theorem is inspired by the universal approximation theorem. The main contribution of the paper is to formulate the theorem and bring one example for the theorem.