There are free lunches

1 INTRODUCTION . No-Free-Lunch ( NFL ) Theorems Wolpert & Macready ( 1997 ) Wolpert ( 2002 ) state that “ all optimization algorithms perform equally well when their performance is averaged across all possible problems ” . Some researchers Alabert et al . ( 2015 ) argued that the necessary conditions for NFL are too restrictive . In practice , there must be some algorithms that perform better than others . However , no one doubt that no algorithm can optimally solve all tasks . The implications of the NFL Theorems are very disappointing , as an universal algorithm is a dream in the field of Artificial Intelligence ( AI ) . This paper proposes the “ There are free lunches ” ( TAFL ) Theorem . The TAFL Theorem states that some algorithms are optimal for all tasks , in the condition that tasks are given in a specific order . We also conclude that the difficulty of solving a new task can decrease when the number of solved tasks increases . The TAFL Theorem implies a very different way of solving AI problems compared to the NFL Theorems . It is not a matter that algorithms at our hands are not powerful or suitable enough for a specific task . One can use an algorithm of limited intelligence to solve a complex problems step by step . We introduce the necessary notation in Section 2 and provide the TAFL Theorem in Section 3 . We analyse the task order mentioned in the TAFL Theorem and point out we can reduce calls of learning algorithms in Section 4 . An application of the TAFL Theorem is provided in Section 5 . Finally , we discuss the implications of the TAFL Theorem in Section 6 and conclude the paper in 7 . 2 PRELIMINARIES . We restrict attention to supervised learning . But the discussion is also applicable for optimization and search fields . The entire sample space is denoted as D. The distribution space is denoted as F . A distribution from F is a mapping f : x → y . A sampling strategy S samples a set of training samples from f , denoted as d = { ( x0 , y0 ) , ... , ( xm , ym ) } , d ∈ D. We consider the sampling strategy is unique for all tasks . A supervised learning algorithm generates a mapping ( or called a hypothesis ) h : x→ y to approximate the distribution f , in terms of off-training-set error . The training samples d are provided to an algorithm La to generate a set of h. The possibility of the algorithm La generating a specific h on d is denoted as P ( h|d , La ) . No-Free-Lunch Theorems state the total off-training-set error across all possible f is always the same for arbitrary algorithms La and Lb.∑ f Eote ( La|d , f ) = ∑ f Eote ( Lb|d , f ) ( 1 ) The off-training-set error Eote ( La|d , f ) is given in Eq.2 , where I ( · ) is 1 when · is ture , otherwise 0 . Eote ( La|d , f ) = ∑ h ∑ di∈f , di /∈d P ( di ) I ( h ( di ) 6= f ( di ) ) P ( h|d , La ) ( 2 ) 3 THE TAFL THEOREM . We consider that a distribution can be parametrized as f ( : θ ) . For a specific algorithm La , there is a set of distribution { fa ( : θa ) |θa ∈ R } which can be optimally solved by La . The TAFL Theorem is described as below . La and Lb are the optimal algorithms , in terms of off-training-set error , for distribution sets { fa ( : θa ) |θa ∈ R } and { fb ( : θb ) |θb ∈ R } respectively . fa ( : θa ) has the form of y = ga ( ∑ i x iθia + θ b a ) . ga can be any continuous nonconstant function . Given a specific fb ( : θ∗b ) and its samples d ∗ b , we can find a serial of tasks , representing by a serial of sets of samples { d0a , ... , dna } . La is the optimal algorithm for each of dia and if we solve each dia in the provided order , then Eote ( La|dna , fb ( : θ∗b ) ) = Eote ( Lb|d∗b , fb ( : θ∗b ) ) . The proof is given as below . According to the universal approximation theorem Hornik et al . ( 1989 ) , fb ( : θ∗b ) can be arbitrarily accurate approximated by a continuous nonconstant function . As fa ( x : θa ) has the form y = ga ( ∑ i x iθia + θ b a ) , the approximation has the form : y0 = x y1 = { yi1|yi1 = fa ( y0 : θ1 , ia ) , i = 0 , ... , m } . . . yn = { yin|yin = fa ( yn−1 : θn , ia ) , i = 0 , ... , k } fb ( x : θ ∗ b ) = yn ( 3 ) Each mapping f : ym−1 → ym has the form of fa ( x : θa ) , therefore they can be optimally solved by La . But for each f : fm−τ → fm , τ ≥ 2 , there is no guarantee that they have the form of fa ( x : θa ) , therefore they may not be optimally solved by La . As each f : ym−1 → ym has the form of fa ( x : θa ) , they are also tasks generated by the distribution fa ( : θa ) . Hence , if a serial of tasks is provided in a order { f : ym−1 → ym|m = 1 , ... , n } , La can solve fb ( : θ∗b ) just as good as Lb . The above proof does not conflict with the universal approximation theorem . The universal approximation theorem is not about the off-training-set error . Therefore , there is no guarantee that a feedforward network is optimal in terms of off-training-set error for any function . One may misunderstand the serial of tasks is similar to hidden layers of a neural network . They are totally different concepts . Each task in the serial is sampled from the task distribution space . Hence , they can be considered as real applications . For example , one task is a OCR task , and another task is a translation task . But the hidden layers of a neural network are not related to real applications , only the outputs of the whole network are related to real applications . 4 ANALYSIS OF THE TASK ORDER . Next , we analyse the task order mentioned in the TAFL Theorem . We first describe our conclusion as below . La is the optimal algorithm in terms of off-training-set error for a set of distribution { fa ( : θa ) |θa ∈ R } . fa ( : θa ) has the form of y = ga ( ∑ i x iθia + θ b a ) . Given a task of distribution fb ( : θ∗b ) , TO ∗ = { d0∗a , ... , dn∗a } is the correct task order for algorithm La to optimally solve fb ( : θ∗b ) . The task order TO ∗ is a sample among all possible task order samples . For a new given task , we sample a set of task orders TOi randomly or following some distribution until the TO∗ is sampled . For each sampled TOi , we call La to solve each of dia in it . We declare that the number of calls of La to solve each sampled dia until d n∗ a is solved can decrease when the number of previously solved tasks increases . Roughly speaking , we can learn how to combine models generated in previous tasks to solve new tasks to reduce the calls of L. The detailed reasons are listed below . 1 We can use a set M to store models of all solved di . Therefore , if a new task needs to solve a di which model is already in M , we can skip running La on it . 2 We can construct a new kind of task based on M . This kind of task , denoted as abstract tasks , learns a mapping from di to a sub-set of models in M : fA : di → m , m ⊂ M , where m should contain models of the correct task order for solving di . The loss of this mapping is shown in Eq.4 . L ( fA|La , fa , M ) = ∑ di ∑ m Eote ( La|di , fa ( m ( x ) : θa ) ) P ( m|di , fA ) ( 4 ) For a given d∗b , generated by fb ( : θ ∗ b ) , if the model of a mapping fA can recognize it and give a set of models m∗ which guarantees that fb ( : θ∗b ) = fa ( m ∗ ( x ) : θa ) , then La can optimally solve this task directly . According to the TAFL Theorem , we know there must be an algorithm LA which is capable to optimally solve all possible Fabs = { fA , fB , ... } . Each training sample of fA is a task in the distribution space F , and the model of fA is optimized to minimize off-training-set error . Therefore , the model of a single fA can be used to generate m for a set of tasks in F . It means after learning a single task fA , we can decrease the number of running La on a set of tasks in F . Following this idea , we can also construct fA2 : ( di , m ) → mA2 , mA2 ⊂ MA2 , where MA2 stores models generated in solving tasks in the distribution space Fabs . All possible fA2 form a new distribution space Fabs2 . The model of a single fA2 can be used to generate mA2 for a set of tasks fA ∈ Fabs which decreases the number of running algorithm LA on those tasks . Generally speaking , for the model of a task in distribution Fabsi , it can be used to decrease the number of running algorithms LAi−1 on a set of tasks in Fabsi−1 . Please note , according to the TAFL Theorem , we can use the same algorithm for all LAi . 3 We consider situations that fAi ∈ Fabsj or fa ∈ Fabsj for some i , a and j . It means models of a distribution space can be used in another distribution space . We consider two situations . In the first situation , fAi , from the distribution space Fabsi , is a part of a task order TOA j . TOA j is sampled for solving a task fAj that comes from the distribution space Fabsj . In this situation , the model of fAi can be used directly to avoid running an algorithm for solving fAi again . In the second situation , fAi is equal to fAj . Therefore , we can avoid learning fAj and use the model of fAi to solve tasks that fAj are required to solve . In those two situations , the calls of learning algorithms are decreased . The elements m of inputs and outputs of fAi and fAj are from different model sets . One may argue that how fAi ∈ Fabsj is possible . Here we suggest some simple ways . For example , we can put all models in the same set , or we can use an encoding function to map any m to a real-valued embeddingE : m→ r. An example is given in the section.5 which shows a way to allow fAi ∈ Fabsj .

Summary: The paper is going beyond the No free lunch theorem and tries to answer the question: "Is it possible to reorder tasks so that I can learn them better?" The paper claims that there is an optimal sequence of task training, but it doesn't provide an algorithm for finding it. There are no real-world or nontrivial synthetic data experiments. One way of understanding the question of the paper is the following example:  Consider a transformer network. We know empirically that training a task that predicts the next word and then a machine translation gives good results. Is that the optimal order of training these tasks?