グループテスト

Template:Db-a10 統計学や組み合わせ数学において, 何らかの検体を特定する作業を, 検体を個別に調べるのではなく, グループにまとめて調べるという作業に分割するようなやりかたをグループテストとよぶ。1943年にRobert Dorfmanによって最初に研究されて以来, グループテストは比較的新しい応用数学の分野であり、幅広い分野で実用化されており, 現在も活発に研究されている。 グループテストのよく知られている例は、直列に接続されたいくつかの電球の中で, いずれか1つだけが壊れている場合である。目的は、できる限り少ない検査（いくつかの電球を電源に接続すること）で、壊れた電球を見つけることである。単純な方法は、各電球を個別に検査することである。しかし、電球の数が多い場合は、電球をグループにまとめる方がはるかに効率的である。たとえば、電球を2つのグループに分けて, 片方のグループの電球を一度にまとめて接続することにより、壊れた電球がどちらのグループに入っているかを判断でき、たった1回の検査で半数の電球を除外できる。 グループテストを実行するためのスキームには単純なものも複雑なものもあり、ひとつのスキームの中の各段階の検査も異なる場合がある。次の段階の検査法が前の段階の検査結果によって変わるようなスキームは適応的手順(adaptive procedure)と呼ばれる一方, すべての検査法が事前に決まるように設計されたスキームは「非適応的手順」(non-adaptive procedure)と呼ばれる。非適応的手順に含まれる検査のスキームの構造は「プーリング設計」として知られている（訳注: この部分の意味が原文ではわからない）。 グループテストには、統計学、生物学、コンピュータサイエンス、医学、エンジニアリング、サイバーセキュリティなど、多くの応用がある。これらの検査スキームに対する関心は、近年もHuman Genome Projectによって再燃している 。 数学の多くの分野とは異なり、グループテストの起源は、Robert Dorfmanという1人の人物が書いた1つの報告書に遡ることができる。第二次世界大戦中に、アメリカ公衆衛生局とセレクティブ・サービス・システムが, 入隊者の中から梅毒の男性全員を排除する大規模プロジェクトに着手したときにその動機が生じた。梅毒について個人を検査するには、血液サンプルを採取して, それを分析する必要がある。当時、この検査には費用がかかり、すべての兵士を個別に検査することは非常に費用がかかり、非効率的だった。 Supposing there are n {\displaystyle n} soldiers, this method of testing leads to n {\displaystyle n} separate tests. If a large proportion of the people are infected then this method would be reasonable. However, in the more likely case that only a very small proportion of the men are infected, a much more efficient testing scheme can be achieved. The feasibility of a more effective testing scheme hinges on the following property: the soldiers can be pooled into groups, and in each group the blood samples can be combined together. The combined sample can then be tested to check if at least one soldier in the group has syphilis. This is the central idea behind group testing. If one or more of the soldiers in this group has syphilis, then a test is wasted (more tests need to be performed to find which soldier(s) it was). On the other hand, if no one in the pool has syphilis then many tests are saved, since every soldier in that group can be eliminated with just one test. The items that cause a group to test positive are generally called defective items (these are the broken lightbulbs, syphilitic men, etc.). Often, the total number of items is denoted as n {\displaystyle n} and d {\displaystyle d} represents the number of defectives if it is assumed to be known. There are two independent classifications for group-testing problems; every group-testing problem is either adaptive or non-adaptive, and either probabilistic or combinatorial. In probabilistic models, the defective items are assumed to follow some probability distribution and the aim is to minimise the expected number of tests needed to identify the defectiveness of every item. On the other hand, with combinatorial group testing, the goal is to minimise the number of tests needed in a 'worst-case scenario' – that is, create a minmax algorithm – and no knowledge of the distribution of defectives is assumed. The other classification, adaptivity, concerns what information can be used when choosing which items to group into a test. In general, the choice of which items to test can depend on the results of previous tests, as in the above lightbulb problem. An algorithm that proceeds by performing a test, and then using the result (and all past results) to decide which next test to perform, is called adaptive. Conversely, in non-adaptive algorithms, all tests are decided in advance. This idea can be generalised to multistage algorithms, where tests are divided into stages, and every test in the next stage must be decided in advance, with only the knowledge of the results of tests in previous stages. Although adaptive algorithms offer much more freedom in design, it is known that adaptive group-testing algorithms do not improve upon non-adaptive ones by more than a constant factor in the number of tests required to identify the set of defective items. In addition to this, non-adaptive methods are often useful in practice because one can proceed with successive tests without first analysing the results of all previous tests, allowing for the effective distribution of the testing process. There are many ways to extend the problem of group testing. One of the most important is called noisy group testing, and deals with a big assumption of the original problem: that testing is error-free. A group-testing problem is called noisy when there is some chance that the result of a group test is erroneous (e.g. comes out positive when the test contained no defectives). The Bernoulli noise model assumes this probability is some constant, q {\displaystyle q} , but in general it can depend on the true number of defectives in the test and the number of items tested. For example, the effect of dilution can be modelled by saying a positive result is more likely when there are more defectives (or more defectives as a fraction of the number tested), present in the test. A noisy algorithm will always have a non-zero probability of making an error (that is, mislabeling an item). Group testing can be extended by considering scenarios in which there are more than two possible outcomes of a test. For example, a test may have the outcomes 0 , 1 {\displaystyle 0,1} and 2 + {\displaystyle 2^{+}} , corresponding to there being no defectives, a single defective, or an unknown number of defectives larger than one. More generally, it is possible to consider the outcome-set of a test to be 0 , 1 , … , k + {\displaystyle {0,1,\ldots ,k^{+}}} for some k ∈ N {\displaystyle k\in \mathbb {N} } . Another extension is to consider geometric restrictions on which sets can be tested. The above lightbulb problem is an example of this kind of restriction: only bulbs that appear consecutively can be tested. Similarly, the items may be arranged in a circle, or in general, a net, where the tests are available paths on the graph. Another kind of geometric restriction would be on the maximum number of items that can be tested in a group, or the group sizes might have to be even and so on. In a similar way, it may be useful to consider the restriction that any given item can only appear in a certain number of tests. There are endless ways to continue remixing the basic formula of group testing. The following elaborations will give an idea of some of the more exotic variants. In the 'good–mediocre–bad' model, each item is one of 'good', 'mediocre' or 'bad', and the result of a test is the type of the 'worst' item in the group. In threshold group testing, the result of a test is positive if the number of defective items in the group is greater than some threshold value or proportion. Group testing with inhibitors is a variant with applications in molecular biology. Here, there is a third class of items called inhibitors, and the result of a test is positive if it contains at least one defective and no inhibitors. The concept of group testing was first introduced by Robert Dorfman in 1943 in a short report published in the Notes section of Annals of Mathematical Statistics. Dorfman's report – as with all the early work on group testing – focused on the probabilistic problem, and aimed to use the novel idea of group testing to reduce the expected number of tests needed to weed out all syphilitic men in a given pool of soldiers. The method was simple: put the soldiers into groups of a given size, and use individual testing (testing items in groups of size one) on the positive groups to find which were infected. Dorfman tabulated the optimum group sizes for this strategy against the prevalence rate of defectiveness in the population. After 1943, group testing remained largely untouched for a number of years. Then in 1957, Sterrett produced an improvement on Dorfman's procedure. This newer process starts by again performing individual testing on the positive groups, but stopping as soon as a defective is identified. Then, the remaining items in the group are tested together, since it is very likely that none of them are defective. The first thorough treatment of group testing was given by Sobel and Groll in their formative 1959 paper on the subject. They described five new procedures – in addition to generalisations for when the prevalence rate is unknown – and for the most optimal one, they provided an explicit formula for the expected number of tests it would use. The paper also made the connection between group testing and information theory for the first time, as well as discussing several generalisations of the group-testing problem and providing some new applications of the theory. Group testing was first studied in the combinatorial context by Li in 1962, with the introduction of Li’s s {\displaystyle s} -stage algorithm. Li proposed an extension of Dorfman's '2-stage algorithm' to an arbitrary number of stages that required no more than t = e log 2 ⁡ ( e ) d log 2 ⁡ ( n ) {\displaystyle t={\frac {e}{\log _{2}(e)}}d\log _{2}(n)} tests to be guaranteed to find d {\displaystyle d} or fewer defectives among n {\displaystyle n} items. The idea was to remove all the items in negative tests, and divide the remaining items into groups as was done with the initial pool. This was to be done s − 1 {\displaystyle s-1} times before performing individual testing. Combinatorial group testing in general was later studied more fully by Katona in 1973. Katona introduced the matrix representation of non-adaptive group-testing and produced a procedure for finding the defective in the non-adaptive 1-defective case in no more than t = ⌈ log 2 ⁡ ( n ) ⌉ {\displaystyle t=\lceil \log _{2}(n)\rceil } tests, which he also proved to be optimal. In general, finding optimal algorithms for adaptive combinatorial group testing is difficult, and although the computational complexity of group testing has not been determined, it is suspected to be hard in some complexity class. However, an important breakthrough occurred in 1972, with the introduction of the generalised binary-splitting algorithm. The generalised binary-splitting algorithm works by performing a binary search on groups that test positive, and is a simple algorithm that finds a single defective in no more than the information-lower-bound number of tests. In scenarios where there are two or more defectives, the generalised binary-splitting algorithm still produces near-optimal results, requiring at most d − 1 {\displaystyle d-1} tests above the information lower bound where d {\displaystyle d} is the number of defectives. Considerable improvements to this were made in 2013 by Allemann, getting the required number of tests to less than 0.187 d + 0.5 log 2 ⁡ ( d ) + 5.5 {\displaystyle 0.187d+0.5\log _{2}(d)+5.5} above the information lower bound when n / d ≥ 38 {\displaystyle n/d\geq 38} and d ≥ 10 {\displaystyle d\geq 10} . This was achieved by changing the binary search in the binary-splitting algorithm to a complex set of sub-algorithms with overlapping test groups. As such, the problem of adaptive combinatorial group testing – with a known number or upper bound on the number of defectives – has essentially been solved, with little room for further improvement. There is an open question as to when individual testing is minmax. Hu, Hwang and Wang showed in 1981 that individual testing is minmax when n ≤ ⌊ ( 5 d + 1 ) / 2 ⌋ {\displaystyle n\leq \lfloor (5d+1)/2\rfloor } , and that it is not minmax when n > 3 d {\displaystyle n>3d} . It is currently conjectured that this bound is sharp: that is, individual testing is minmax if and only if n ≤ 3 d {\displaystyle n\leq 3d} . Some progress was made in 2000 by Ricccio and Colbourn, who showed that for large n {\displaystyle n} , individual testing is minmax when d ≥ n / log 3 / 2 ⁡ ( 3 ) ≈ 0.369 n {\displaystyle d\geq n/\log _{3/2}(3)\approx 0.369n} . One of the key insights in non-adaptive group testing is that significant gains can be made by eliminating the requirement that the group-testing procedure be certain to succeed (the "combinatorial" problem), but rather permit it to have some low but non-zero probability of mis-labelling each item (the "probabilistic" problem). It is known that as the number of defective items approaches the total number of items, exact combinatorial solutions require significantly more tests than probabilistic solutions — even probabilistic solutions permitting only an asymptotically small probability of error. In this vein, Chan et al. (2011) introduced COMP, a probabilistic algorithm that requires no more than t = e d ( 1 + δ ) ln ⁡ ( n ) {\displaystyle t=ed(1+\delta )\ln(n)} tests to find up to d {\displaystyle d} defectives in n {\displaystyle n} items with a probability of error no more than n − δ {\displaystyle n^{-\delta }} . This is within a constant factor of the t = O ( d log 2 ⁡ n ) {\displaystyle t=O(d\log _{2}n)} lower bound. Chan et al. (2011) also provided a generalisation of COMP to a simple noisy model, and similarly produced an explicit performance bound, which was again only a constant (dependent on the likelihood of a failed test) above the corresponding lower bound. In general, the number of tests required in the Bernoulli noise case is a constant factor larger than in the noiseless case. Aldridge, Baldassini and Johnson (2014) produced an extension of the COMP algorithm that added additional post-processing steps. They showed that the performance of this new algorithm, called DD, strictly exceeds that of COMP, and that DD is 'essentially optimal' in scenarios where d 2 ≥ n {\displaystyle d^{2}\geq n} , by comparing it to a hypothetical algorithm that defines a reasonable optimum. The performance of this hypothetical algorithm suggests that there is room for improvement when d 2 < n {\displaystyle d^{2}<n} , as well as suggesting how much improvement this might be. This section formally defines the notions and terms relating to group testing. The input vector, x = ( x 1 , x 2 , … , x n ) {\displaystyle \mathbf {x} =(x_{1},x_{2},\dots ,x_{n})} , is defined to be a binary vector of length n {\displaystyle n} (that is, x ∈ { 0 , 1 } n {\displaystyle \mathbf {x} \in \{0,1\}^{n}} ), with the j-th item being called defective if and only if x j = 1 {\displaystyle x_{j}=1} . Further, any non-defective item is referred called a 'good' item. x {\displaystyle \mathbf {x} } is intended to describe the (unknown) set of defective items. The key property of x {\displaystyle \mathbf {x} } is that it is an implicit input. That is to say, there is no direct knowledge of what the entries of x {\displaystyle \mathbf {x} } are, other than that which can be inferred via some series of 'tests'. This leads on to the next definition. Let x {\displaystyle \mathbf {x} } be an input vector. A set, S ⊆ { 1 , 2 , … , n } {\displaystyle S\subseteq \{1,2,\dots ,n\}} is called a test. When testing is noiseless, the result of a test is positive when there exists j ∈ S {\displaystyle j\in S} such that x j = 1 {\displaystyle x_{j}=1} , and the result is negative otherwise. Therefore, the goal of group testing is to come up with a method for choosing a 'short' series of tests that allow x {\displaystyle \mathbf {x} } to be determined, either exactly or with a high degree of certainty. A group-testing algorithm is said to make an error if it incorrectly labels an item (that is, labels any defective item as non-defective or vice versa). This is not the same thing as the result of a group test being incorrect. An algorithm is called zero-error if the probability that it makes an error is zero. t ( d , n ) {\displaystyle t(d,n)} denotes the minimum number of tests required to always find d {\displaystyle d} defectives among n {\displaystyle n} items with zero probability of error by any group-testing algorithm. For the same quantity but with the restriction that the algorithm is non-adaptive, the notation t ¯ ( d , n ) {\displaystyle {\bar {t}}(d,n)} is used. Since it is always possible to resort to individual testing by setting S j = { j } {\displaystyle S_{j}=\{j\}} for each 1 ≤ j ≤ n {\displaystyle 1\leq j\leq n} , it must be that that t ¯ ( d , n ) ≤ n {\displaystyle {\bar {t}}(d,n)\leq n} . Also, since any non-adaptive testing procedure can be written as an adaptive algorithm by simply performing all the tests without regard to their outcome, t ( d , n ) ≤ t ¯ ( d , n ) {\displaystyle t(d,n)\leq {\bar {t}}(d,n)} . Finally, when 0 ≠ d ≠ n {\displaystyle 0\neq d\neq n} , there is at least one item whose defectiveness must be determined (by at least one test), and so 1 ≤ t ( d , n ) {\displaystyle 1\leq t(d,n)} . In summary (when assuming 0 ≠ d ≠ n {\displaystyle 0\neq d\neq n} ), 1 ≤ t ( d , n ) ≤ t ¯ ( d , n ) ≤ n {\displaystyle 1\leq t(d,n)\leq {\bar {t}}(d,n)\leq n} . A lower bound on the number of tests needed can be described using the notion of sample space, denoted S {\displaystyle {\mathcal {S}}} , which is simply the set of possible placements of defectives. For any group testing problem with sample space S {\displaystyle {\mathcal {S}}} and any group-testing algorithm, it can be shown that t ≥ ⌈ log 2 ⁡ | S | ⌉ {\displaystyle t\geq \lceil \log _{2}{|{\mathcal {S}}|}\rceil } , where t {\displaystyle t} is the minimum number of tests required to identify all defectives with a zero probability of error. This is called the information lower bound. This bound is derived from the fact that after each test, S {\displaystyle {\mathcal {S}}} is split into two disjoint subsets, each corresponding to one of the two possible outcomes of the test. However, the information lower bound itself is usually unachievable, even for small problems. This is because the splitting of S {\displaystyle {\mathcal {S}}} is not arbitrary, since it must be realisable by some test. In fact, the information lower bound can be generalised to the case where there is a non-zero probability that the algorithm makes an error. In this form, the theorem gives us an upper bound on the probability of success based on the number of tests. For any group-testing algorithm that performs t {\displaystyle t} tests, the probability of success, P ( success ) {\displaystyle \mathbb {P} ({\textrm {success}})} , satisfies P ( success ) ≤ t / log 2 ⁡ ( n d ) {\displaystyle \mathbb {P} ({\textrm {success}})\leq t/\log _{2}{n \choose d}} . This can be strengthened to: P ( success ) ≤ 2 t ( n d ) {\displaystyle \mathbb {P} ({\textrm {success}})\leq {\frac {2^{t}}{n \choose d}}} . Algorithms for non-adaptive group testing consist of two distinct phases. First, it is decided how many tests to perform and which items to include in each test. In the second phase, often called the decoding step, the results of each group test are analysed to determine which items are likely to be defective. The first phase is usually encoded in a matrix as follows. Suppose a non-adaptive group testing procedure for n {\displaystyle n} items consists of the tests S 1 , S 2 , … , S t {\displaystyle S_{1},S_{2},\dots ,S_{t}} for some t ∈ N ≥ 0 {\displaystyle t\in \mathbb {N} _{\geq 0}} . The testing matrix for this scheme is the t × n {\displaystyle t\times n} binary matrix, M {\displaystyle M} , where ( M ) i j = 1 {\displaystyle (M)_{ij}=1} if and only if j ∈ S i {\displaystyle j\in S_{i}} (and is zero otherwise). Thus each column of M {\displaystyle M} represents an item and each row represents a test, with a 1 {\displaystyle 1} in the ( i , j ) -th {\displaystyle (i,j){\textrm {-th}}} entry indicating that the i -th {\displaystyle i{\textrm {-th}}} test included the j -th {\displaystyle j{\textrm {-th}}} item and a 0 {\displaystyle 0} indicating otherwise. As well as the vector x {\displaystyle \mathbf {x} } (of length n {\displaystyle n} ) that describes the unknown defective set, it is common to introduce the result vector, which describes the results of each test. Let t {\displaystyle t} be the number of tests performed by a non-adaptive algorithm. The result vector, y = ( y 1 , y 2 , … , y t ) {\displaystyle \mathbf {y} =(y_{1},y_{2},\dots ,y_{t})} , is a binary vector of length t {\displaystyle t} (that is, y ∈ { 0 , 1 } t {\displaystyle \mathbf {y} \in \{0,1\}^{t}} ) such that y i = 1 {\displaystyle y_{i}=1} if and only if the result of the i -th {\displaystyle i{\textrm {-th}}} test was positive (i.e. contained at least one defective). With these definitions, the non-adaptive problem can be reframed as follows: first a testing matrix is chosen, M {\displaystyle M} , after which the vector y {\displaystyle \mathbf {y} } is returned. Then the problem is to analyse y {\displaystyle \mathbf {y} } to find some estimate for x {\displaystyle \mathbf {x} } . In the simplest noisy case, where there is a constant probability, q {\displaystyle q} , that a group test will have an erroneous result, one considers a random binary vector, v {\displaystyle \mathbf {v} } , where each entry has a probability q {\displaystyle q} of being 1 {\displaystyle 1} , and is 0 {\displaystyle 0} otherwise. The vector that is returned is then y