Office Action 2: Notice of Allowance (2024-08-28)

CLAIMS:
AMENDMENTS TO CLAIMSThe following listings of claims replaces all prior versions and listings of claims in the present application:

1\. (Currently Amended) An apparatus comprising at least one processor and at least one non-transitory memory comprising program code stored thereon, wherein the at least one non-transitory memory and the program code are configured to, with the at least one processor, cause the apparatus to at least:
transform an original training data set corresponding to a multi-class classification task into at least two adapted training data sets each corresponding to a binary classification task, the at least two adapted training data sets including a coarse-level adapted training data set and at least one fine-level adapted training data set, wherein a coarse-level instance of a binary classification model configured to perform the binary classification task is trained based at least in part on the coarse-level adapted training data set, and at least one fine-level instance of the binary classification model is trained based at least in part on, respectively, each at least one fine-level adapted training data set, wherein the original training data set comprises a multi-class classification of training objects each into an original classification set of at least three original classification sets, and wherein the coarse-level adapted training data set comprises a binary classification of the training objects from the original training data set each into one of:a majority classification set corresponding to a particular original classification set of the at least three original classification sets into which a largest proportion of the training objects are classified in the original training data set relative to other original classification sets of the at least three original classification sets; and
a non-majority classification set representing a combination of the other original classification sets of the at least three original classification sets other than the particular original classification set to which the majority classification set corresponds;
transform binary output data generated with respect to an input data set using the trained coarse-level instance of the binary classification model and the trained at least one

fine-level instance of the binary classification model into multi-class output data corresponding to the multiclass classification task; andcause performance of at least one enterprise management operation based at least in part on the multi-class output data.

2\. (Canceled)

3\. (Canceled)

4\. (Currently Amended) The apparatus of claim \[\[3\]\]1, wherein the at least one fine- level adapted training data set each comprises a binary classification of a selected subset of the training objects, including only training objects classified into the non-majority classification set in the coarse-level adapted training data set, each into one of a first classification set and a second classification set, each of which corresponds to a different original classification set from among the combination of original classification sets represented by the non-majority classification set of the coarse-level adapted training data set.

5\. (Currently Amended) The apparatus of claim \[\[3\]\]1, wherein a coarse-level portion of the binary output data, which coarse-level portion is generated using the trained coarse- level instance of the binary classification model, comprises a binary classification of all production objects from the input data set each into one of: the majority classification set and the non-majority classification set.

6\. (Original) The apparatus of claim 5, wherein a fine-level portion of the binary output data, which fine-level portion is generated using the trained at least one fine-level instance of the binary classification model, comprises a binary classification of a selected subset of the production objects including only production objects classified into the non-majority classification set in the coarse-level portion of the binary output data each into a particular original classification set from among the combination of original classification sets represented by the non-majority classification set.

7\. (Original) The apparatus of claim 1, wherein the at least one non-transitory memory and the program code are configured to, with the at least one processor, further cause the apparatus to at least:
train the coarse-level instance of the binary classification model based at least in part on the coarse-level adapted training data set; and
train each at least one fine-level instance of the binary classification model based at least in part on, respectively, each at least one fine-level adapted training data set.

8\. (Original) The apparatus of claim 1, wherein the at least one non-transitory memory and the program code are configured to, with the at least one processor, further cause the apparatus to at least:
generate the binary output data with respect to the input data set using the trained coarse-level instance of the binary classification model and the trained at least one fine-level instance of the binary classification model.

9\. (Original) The apparatus of claim 8, wherein generating the binary output data and transforming the binary output data into the multi-class output data comprises:
generating a coarse-level portion of the binary output data by performing, using the trained coarse-level instance of the binary classification model, a coarse-level classification of all production objects from the input data set into one of the majority classification set and the non-majority classification set;
generating a fine-level portion of the binary output data by performing, using the trained at least one fine-level instance of the binary classification model, a fine-level classification of a selected subset of the production objects including only production objects classified into the non-majority classification set in the coarse-level portion of the binary output data; and
combining the coarse-level portion of the binary output data and the fine-level portion of the binary output data to produce the multi-class output data.

10\. (Original) The apparatus of claim 1, wherein the multi-class classification task includes classification of text objects containing text characterizing quality management

events associated with products produced by an enterprise into classification sets corresponding to predicted levels of severity of the quality management events.

11\. (Currently Amended) A computer-implemented method comprising:
transforming an original training data set corresponding to a multi-class classification task into at least two adapted training data sets each corresponding to a binary classification task, the at least two adapted training data sets including a coarse-level adapted training data set and at least one fine-level adapted training data set, wherein a coarse-level instance of a binary classification model configured to perform the binary classification task is trained based at least in part on the coarse-level adapted training data set, and at least one fine-level instance of the binary classification model is trained based at least in part on, respectively, each at least one fine-level adapted training data set, wherein the original training data set comprises amulti-class classification of training objects each into an original classification set of at least three original classification sets, and wherein the coarse-level adapted training data set comprises a binary classification of the training objects from the original training data set each into one of:a majority classification set corresponding to a particular original classification set of the at least three original classification sets into which a largest proportion of the training objects are classified in the original training data set relative to other original classification sets of the at least three original classification sets; and
a non-majority classification set representing a combination of the other original classification sets of the at least three original classification sets other than the particular original classification set to which the majority classification set corresponds;
transforming binary output data generated with respect to an input data set using the trained coarse-level instance of the binary classification model and the trained at least one fine-level instance of the binary classification model into multi-class output data corresponding to the multiclass classification task; and
causing performance of at least one enterprise management operation based at least in part on the multi-class output data.

12\. (Canceled)

13\. (Canceled)

14\. (Currently Amended) The method of claim \[\[13\]\]11, wherein the at least one fine- level adapted training data set each comprises a binary classification of a selected subset of the training objects, including only training objects classified into the non-majority classification set in the coarse-level adapted training data set, each into one of a first classification set and a second classification set, each of which corresponds to a different original classification set from among the combination of original classification sets represented by the non-majority classification set of the coarse-level adapted training data set.

15\. (Currently Amended) The method of claim \[\[13\]\]11, wherein a coarse-level portion of the binary output data, which coarse-level portion is generated using the trained coarse- level instance of the binary classification model, comprises a binary classification of all production objects from the input data set each into one of: the majority classification set and the non-majority classification set.

16\. (Original) The method of claim 15, wherein a fine-level portion of the binary output data, which fine-level portion is generated using the trained at least one fine-level instance of the binary classification model, comprises a binary classification of a selected subset of the production objects including only production objects classified into the non- majority classification set in the coarse-level portion of the binary output data each into a particular original classification set from among the combination of original classification sets represented by the non-majority classification set.

17\. (Original) The method of claim 11, further comprising:
training the coarse-level instance of the binary classification model based at least in part on the coarse-level adapted training data set; and
training each at least one fine-level instance of the binary classification model based at least in part on, respectively, each at least one fine-level adapted training data set.

18\. (Original) The method of claim 11, further comprising generating the binary output data with respect to the input data set using the trained coarse-level instance of the binary classification model and the trained at least one fine-level instance of the binary classification model.

19\. (Original) The method of claim 18, wherein generating the binary output data and transforming the binary output data into the multi-class output data comprises:
generating a coarse-level portion of the binary output data by performing, using the trained coarse-level instance of the binary classification model, a coarse-level classification of all production objects from the input data set into one of the majority classification set and the non-majority classification set;
generating a fine-level portion of the binary output data by performing, using the trained at least one fine-level instance of the binary classification model, a fine-level classification of a selected subset of the production objects including only production objects classified into the non-majority classification set in the coarse-level portion of the binary output data; and
combining the coarse-level portion of the binary output data and the fine-level portion of the binary output data to produce the multi-class output data.

20\. (Currently Amended) A computer program product comprising at least one non- transitory computer-readable storage medium having computer-readable program code portions stored therein, the computer-readable program code portions comprising an executable portion configured to:
transform an original training data set corresponding to a multi-class classification task into at least two adapted training data sets each corresponding to a binary classification task, the at least two adapted training data sets including a coarse-level adapted training data set and at least one fine-level adapted training data set, wherein a coarse-level instance of a binary classification model configured to perform the binary classification task is trained based at least in part on the coarse-level adapted training data set and at least one fine-level instance of the binary classification model is trained based at least in part on, respectively, each at

least one fine-level adapted training data set, wherein the original training data set comprises a multi-class classification of training objects each into an original classification set of at least three original classification sets, and wherein the coarse-level adapted training data set comprises a binary classification of the training objects from the original training data set each into one of:a majority classification set corresponding to a particular original classification set of the at least three original classification sets into which a largest proportion of the training objects are classified in the original training data set relative to other original classification sets of the at least three original classification sets; and
a non-majority classification set representing a combination of the other original classification sets of the at least three original classification sets other than the particular original classification set to which the majority classification set corresponds;
transform binary output data generated with respect to an input data set using the trained coarse-level instance of the binary classification model and the trained at least one fine-level instance of the binary classification model into multi-class output data corresponding to the multiclass classification task; and
cause performance of at least one enterprise management operation based at least in part on the multi-class output data.

OFFICE ACTION:
DETAILED ACTION
Notice of Pre-AIA  or AIA  Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status
Claims 1, 4-11, and 14-20 are allowed in this Office action.
Examiner’s Comment / Summary of Related Prior Art
The prior art on record are summarized as follows:
Ezhov et al. (Pub. No. US 2022/0122261) teaches automated segmentation of volumetric images comprising the steps of: receiving a coarse volumetric image comprising a jaw/tooth structure in terms of voxels; defining each voxel a distinct anatomical identifier based on a probabilistic distribution for each of a tooth/non-tooth class; applying the defined coarse image through a coarse model for a coarse output; combining coarse output with the coarse volumetric image; and generating a probabilistic segmentation from applying the combined coarse output/image through a fine model; and converting the probabilistic to a polygonal mesh for each defined class applying a volume-to-mesh algorithm.
Nicacio Braga (Pub. No. US 2023/0139437) teaches a training batch of data points from training data for a plurality of classes and builds a multi-class classifier having a series of binary classifiers arranged in a first order. Each of the binary classifiers is associated with a respective class. The embodiment trains the multi-class classifier with the binary classifiers arranged in a first order and, at each binary classifier, the embodiment identifies data points as belonging to the class associated with the respective classifier and updates the training batch to exclude the classified data points. The multi-class classifier is modified by changing the order of classifiers and repeats the training of the multi-class classifier with the series of binary classifiers arranged in a second order. A final configuration of the multi-class classifier is selected based at least in part on a comparison of first training results to the second training results.
Rapowitz et al. (Pub. No. US 2023/0342691) teaches training and deploying a machine learning model to generate an assessment of risks and mitigations in response to a novel initiative request. After generating labeled data from a corpus of prior risk assessments, a machine learning model may be trained to programmatically generate a risk assessment in response to a novel initiative request. A risk auditor may subsequently review the risk assessment generated using the machine learning model to provide various feedback reflecting the accuracy of the risks and mitigations. The machine learning model may then be retrained based on the feedback provided by the risk auditor to provide more accurate risks and mitigations in response to future initiatives.
Kaufman et al. (Pub. No. US 2022/0237801) teaches generating a multiclass image segmentation model(s) can include receiving multiple single-class image datasets, receiving a target mask for each of the single-class image datasets, receiving a condition of an object associated with each of the single-class image datasets, and generating the multiclass image segmentation model(s) based on the single-class image datasets, the target masks, and the identification of the target objects.
Lee et al. (Pub. No. US 2020/0193153) teaches balancing the selection of high- quality matches for each input text segment, so that a stored text segment with a high similarity score is chosen for each input text segment, with the desire for the stored text segments to come from a single or relatively smaller number of reference documents. The combination generation algorithm used is an optimization algorithm that optimizes across two or more variables, where for example one of the variables is the text similarity scores for the individual input text segments and another is the number of reference documents used.
Sharma et al. (Pub. No. US 2020/0356813) teaches a new data layer, being used to avoid bias in training process by favoring the gradients' updates towards the class with the majority of the samples, with the concept of weighted buckets as class containers. Each bucket has three properties: the source file of its contents, the unique label that represents its contents, and the weight that determines the likelihood with which a sample from the bucket might get picked as a member of each training batch. In this design, samples of a single class may be represented using several buckets with the same label, and if desired different weights, but each bucket is not capable of holding more than a single class. Each bucket's weight enables us to uniformly represent classes for which we have different numbers of training images, by assigning equal weights to their corresponding buckets.
Reasons for Allowance
The following is an examiner's statement of reasons for allowance of Claims 1, 4-11, and 14-20:
In interpreting the current set of claims, in view of Applicant’s latest remarks, and in light of the search/examination and the available prior art, the Examiner finds the claimed invention to be patentably distinct from the prior art of records. Specifically, the prior art of records, individually or in combination, fail to explicitly teach, suggest or render obvious the claimed invention as recited in independent claims 1, 11, and 20.
Other dependent claims are also allowed based on their dependencies on claims 1, 11, and 20.
Any comments considered necessary by the Applicant must be submitted no later than the payment of the issue fee and, to avoid processing delays, should preferably accompany the issue fee. Such submissions should be clearly labeled “Comments on Statement of Reasons for Allowance.”
Contact Information

APPLICANT REBUTTAL: REMARKS/ARGUMENTS
In response to the Office Action, claims 1, 4, 5, 11, 14, 15, and 20 have been herein amended. Claims 2, 3, 12, and 13 are canceled herein without any prejudice or disclaimer. Support for the amendments can be found at least, for example, at allowable dependent claim 3 and at intervening dependent claim 2 as previously presented. No new subject matter has been added. Thus, claims 1, 4-11, and 14-20 are pending in the present application. The Applicant respectfully requests reconsideration of this application.
Claim Rejection(s)
Claims 1-2, 7-9, 11-12, and 17-20 stand rejected under 35 U.S.C. § 103 as being unpatentable over US Patent Application Publication No. US 2022/0122261 Al (hereinafter "Ezhov") in view of US Patent Application Publication No. US 2023/0139437 Al (hereinafter "Braga").
Claim 10 stands rejected under 35 U.S.C. § 103 as being unpatentable over Ezhov in view of Braga and further in view of US Patent Application Publication No. US 2023/0342691 Al (hereinafter "Rapowitz").
ALLOWABLE SUBJECT MATTER
The Applicant would like to thank the Office for indicating that claims 3-6 and 13-16 would be allowable if rewritten in independent form to incorporate the limitations of the base claims and all intervening claim(s). Accordingly, the Applicant has amended independent claim 1 to incorporate features of allowable dependent claim 3 and intervening claim 2. Independent claims 11 and 20 have been amended similar to independent claim 1.
REJECTION UNDER 35 U.S.C.
§
Claims 1-2, 7-9, 11-12, and 17-20 stand rejected under 35 U.S.C. § 103 as being unpatentable over US Patent Application Publication No. US 2022/0122261 Al (hereinafter "Ezhov") in view of US Patent Application Publication No. US 2023/0139437 Al (hereinafter "Braga").
The Applicant respectfully submits that independent claim 1 has been amended to incorporate features of allowable dependent claim 3 and intervening claim 2. Independent claims
11 and 20 have been amended similar to independent claim 1. Thus, the Applicant respectfully requests that the rejection of claims 1, 7-9, 11, and 17-20 under 35 U.S.C. § 103 be now withdrawn.
Claim 10 stands rejected under 35 U.S.C. § 103 as being unpatentable over Ezhov in view of Braga and further in view of US Patent Application Publication No. US 2023/0342691 Al (hereinafter "Rapowitz").
The Applicant submits that dependent claim 10 is also allowable based at least on its dependence on amended independent claim 1. Therefore, the Applicant respectfully requests that the rejection of claim 10 under 35 U.S.C. § 103 be withdrawn.
CONCLUSION


CITED REFERENCE SUMMARIES: Ref US20200356813A1: Claims:
- 16 . (canceled)
A neural network comprising: one or more input stages for receiving image data; one or more intermediate stages, cascaded following said one or more input stages, each of said intermediate stages having an input coupled to an output of a previous stage; and plural concluding stages, each having an input coupled to an output of an intermediate stage; wherein a first of said plural concluding stages has N output neurons, and a second of said plural concluding stages has M output neurons.
one or more input stages for receiving image data;
one or more intermediate stages, cascaded following said one or more input stages, each of said intermediate stages having an input coupled to an output of a previous stage; and
plural concluding stages, each having an input coupled to an output of an intermediate stage;
wherein a first of said plural concluding stages has N output neurons, and a second of said plural concluding stages has M output neurons.
The neural network of claim 17 in which M is less than N.
The neural network of claim 17 in which the first concluding stage comprises plural layers.
The neural network of claim 17 in which a first neuron in a first of said concluding stages, and a first neuron in a second of said concluding stages, are each configured to fire when an image depicting a first particular grocery item is presented to the one or more input stages.
The neural network of claim 20 wherein the particular grocery item is identified not by a single neuron in said first and second concluding stages that fires with a greatest output signal, but rather by a combination of (a) said first neuron in the first concluding stage, which fires with an output signal greater than all other neurons in the first concluding stage, together with (b) said first neuron in the second concluding stage, which fires with an output signal greater than all other neurons in the second concluding stage.
The neural network of claim 20 in which a second neuron in the first concluding stage, and a second neuron in the second concluding stage, are each configured to fire when an image depicting a second particular grocery item is presented to the one or more input stages, wherein the first and second particular grocery items are two brandmates of a particular type of product, distinguished as lemon and lime variants, or regular and low-sodium variants, or seeded and unseeded variants, of said particular type of product.
The neural network of claim 17 in which at least one of said concluding stages comprises a convolutional stage.
A method comprising the acts: providing a neural network including one or more input stages for receiving image data; one or more intermediate stages, cascaded following said one or more input stages, each of said intermediate stages having an input coupled to an output of a previous stage; and plural concluding stages, each having an input coupled to an output of an intermediate stage; wherein a first of said plural concluding stages has N output neurons, and a second of said plural concluding stages has M output neurons, where M is less than N; training the network to respond to images depicting legacy grocery products by firing neurons in the first concluding stage; and training the network to respond to images depicting newer grocery products by firing neurons in the second concluding stage; the method further included retraining the network when items are added to a grocery's offerings, said retraining comprising retraining the second, but not the first, of said concluding stages.
providing a neural network including one or more input stages for receiving image data; one or more intermediate stages, cascaded following said one or more input stages, each of said intermediate stages having an input coupled to an output of a previous stage; and plural concluding stages, each having an input coupled to an output of an intermediate stage; wherein a first of said plural concluding stages has N output neurons, and a second of said plural concluding stages has M output neurons, where M is less than N;
training the network to respond to images depicting legacy grocery products by firing neurons in the first concluding stage; and
training the network to respond to images depicting newer grocery products by firing neurons in the second concluding stage;
the method further included retraining the network when items are added to a grocery's offerings, said retraining comprising retraining the second, but not the first, of said concluding stages.
A method comprising the acts: providing a neural network including one or more input stages for receiving image data; one or more intermediate stages, cascaded following said one or more input stages, each of said intermediate stages having an input coupled to an output of a previous stage; and plural concluding stages, each having an input coupled to an output of an intermediate stage; wherein a first of said plural concluding stages has N output neurons, and a second of said plural concluding stages has M output neurons, where M is less than N; training the network to respond to images depicting first and second grocery products by firing first and second neurons, respectively, in the first concluding stage; and training the network to respond to images depicting said first and second grocery products by firing first and second neurons, respectively, in the second concluding stage.
training the network to respond to images depicting first and second grocery products by firing first and second neurons, respectively, in the first concluding stage; and
training the network to respond to images depicting said first and second grocery products by firing first and second neurons, respectively, in the second concluding stage.
The method of claim 25 wherein the first and second grocery items are two brandmates of a particular type of product.
The method of claim 26 wherein the first and second grocery items are brandmates distinguished as flavor variants or ingredient variants, or regular and low-sodium variants, or seeded and unseeded variants, of said particular type of product.
The method of claim 27 in which the first and second grocery items are brandmates distinguished as flavor variants, namely lemon or lime variants.
The method of claim 27 in which the first and second grocery items are brandmates distinguished as ingredient variants, namely regular sodium or low sodium variants.
The method of claim 27 in which the first and second grocery items are brandmates distinguished as ingredient variants, namely seeded or seedless variants.
- 63 . (canceled)
Ref US20220122261: Claims:
1. A method for automated segmentation of volumetric images, said method comprising the steps of:
receiving a volumetric image comprising a jaw/tooth structure in terms of voxels; and
defining each voxel a distinct anatomical structure by identifier based on a probabilistic distribution for each of the anatomical structure.
2. The method of claim 1, wherein the at least one received volumetric image comprises a 3-D pixel array.
3. The method of claim 2, further comprising the step of pre-processing by converting the 3-D pixel array into an array of Hounsfield Unit (HU) radio intensity measurements.
4. The method of claim 1, further comprising the step of pre-processing the received image by rescaling using linear interpolation.
5. The method of claim 4, wherein the pre-processing comprises using any one of a normalization schemes to account for variations in image value intensity depending on at least one of an input or output of volumetric image.
6. The method of claim 1, further comprising the step of localizing using a V-Net-based fully convolutional neural network.
7. The method of claim 1, further comprising the step of extracting anatomical structure by finding a minimum bounding rectangle around the defined structure.
8. The method of claim 7, wherein the bounding rectangle extends by at least 15 mm vertically and 8 mm horizontally (equally in all directions) to capture the tooth and surrounding context.
9. The method of claim 1, further comprising the step of condition classification using a DenseNet 3-D convolutional neural network.
10. The method of claim 1, further comprising converting the probabilistic to a polygonal mesh for each defined class applying a volume-to-mesh algorithm.
11. A method for automated segmentation of volumetric images, said method comprising the steps of:
receiving a volumetric image comprising a jaw/tooth structure in terms of voxels; defining each voxel a distinct anatomical identifier based on a probabilistic distribution for each of an anatomical structure; and converting the probabilistic to a polygonal mesh for each defined class applying a volume-to-mesh algorithm.
12. The method of claim 11, wherein the at least one received volumetric image comprises a 3-D pixel array.
13. The method of claim 11, further comprising defining voxels using a V-Net-based fully convolutional neural network.
14. The method of claim 11, further comprising the step of extracting an anatomical structure by finding a minimum bounding rectangle around the localized and identified tooth/non-tooth.
15. The method of claim 14, wherein the bounding rectangle extends by at least 15 mm vertically and 8 mm horizontally (equally in all directions) to capture the tooth and surrounding context.
16. The method of claim 11, further comprising the step of condition classification using a DenseNet 3-D convolutional neural network.
17. A method for automated segmentation of volumetric images, said method comprising the steps of:
receiving a coarse volumetric image comprising a jaw/tooth structure in terms of voxels; defining each voxel a distinct anatomical identifier based on a probabilistic distribution for each of an anatomical structure; applying the defined coarse image through a coarse model for a coarse output; combining coarse output with the coarse volumetric image; and generating a probabilistic segmentation from applying the combined coarse output/image through a fine model; and converting the probabilistic to a polygonal mesh for each defined class applying a volume-to-mesh algorithm.
18. The method of claim 17, further comprising segmenting the polygonal mesh for each defined class.
19. The method of claim 17, further comprising a patch-based approach for the fine model segmentation.
20. The method of claim 17, further comprising a Region-of-Interest (RoI) approach for the fine model segmentation.
Ref US20230139437: Claims:
1. A computer implemented method comprising:
generating a training batch of data points from training data associated with a plurality of classes; building a multi-class classifier having a series of binary classifiers arranged in a first order, wherein each of the series of binary classifiers is associated with a respective class of the plurality of classes; training the multi-class classifier with the series of binary classifiers being arranged in the first order using the training batch of data points, the training comprising, at each binary classifier in succession:
identifying data points as belonging to the associated class resulting in classified data points; and
updating the training batch to exclude the classified data points prior to the training batch being inputted into a subsequent binary classifier;
 modifying the multi-class classifier by changing the order of the series of binary classifiers resulting in the series of binary classifiers being arranged in a second order; repeating the training of the multi-class classifier with the series of binary classifiers being arranged in the second order using the training batch of data points; and selecting a final configuration of the multi-class classifier based at least in part on a comparison of first training results from training with the series of binary classifiers arranged in the first order to second training results from training with the series of binary classifiers arranged in the second order.
2. The method of claim 1, wherein the generating of the training batch of data points comprises randomly selecting the data points from the training data.
3. The method of claim 1, wherein each of the series of binary classifiers outputs a first value for data points predicted to be part of the class associated with the binary classifier and outputs a second value for data points predicted to be part of any other classes of the plurality of classes.
4. The method of claim 1, wherein training of the multi-class classifier further comprises applying a multi-class voting classifier to any data points remaining unclassified after progressing through the series of binary classifiers.
5. The method of claim 1, further comprising:
generating a first random order as the first order of arrangement of the series of binary classifiers; and generating a second random order as the second order of arrangement of the series of binary classifiers.
6. The method of claim 1, wherein the modifying of the multi-class classifier comprises:
designating a plurality of groups of binary classifiers from the series of binary classifiers such that each of the binary classifiers is in exactly one of the plurality of groups; and arranging the binary classifiers in each of the groups according to respective random orders of arrangement.
7. The method of claim 6, further comprising:
modifying the multi-class classifier by changing the order of the series of binary classifiers resulting in the series of binary classifiers being arranged in a third order; and repeating the training of the multi-class classifier with the series of binary classifiers being arranged in the third order using the training batch of data points.
8. The method of claim 7, wherein the modifying of the multi-class classifier resulting in the series of binary classifiers being arranged in the third order comprises:
designating a plurality of sub-groups of binary classifiers from each of the groups of binary classifiers such that each of the binary classifiers is in exactly one of the plurality of sub-groups; and arranging the binary classifiers in each of the sub-groups according to respective random orders of arrangement.
9. The method of claim 8, wherein each of the groups of binary classifiers includes an equal number of the sub-groups.
10. The method of claim 1, further comprising:
generating a first overall confidence score based on cumulative confidence scores from classifying each of the data points with the binary classifiers arranged in the first order; and generating second overall confidence score based on cumulative confidence scores from classifying each of the data points with the binary classifiers arranged in the second order.
11. The method of claim 10, wherein the comparison of the first training results to the second training results comprises comparing the first overall confidence score to the second overall confidence score.
12. A computer program product, the computer program product comprising one or more computer readable storage media, and program instructions collectively stored on the one or more computer readable storage media, the program instructions executable by one or more processors to cause the one or more processors to perform operations comprising:
generating a training batch of data points from training data associated with a plurality of classes; building a multi-class classifier having a series of binary classifiers arranged in a first order, wherein each of the series of binary classifiers is associated with a respective class of the plurality of classes; training the multi-class classifier with the series of binary classifiers being arranged in the first order using the training batch of data points, the training comprising, at each binary classifier in succession:
identifying data points as belonging to the associated class resulting in classified data points; and
updating the training batch to exclude the classified data points prior to the training batch being inputted into a subsequent binary classifier;
 modifying the multi-class classifier by changing the order of the series of binary classifiers resulting in the series of binary classifiers being arranged in a second order; repeating the training of the multi-class classifier with the series of binary classifiers being arranged in the second order using the training batch of data points; and selecting a final configuration of the multi-class classifier based at least in part on a comparison of first training results from training with the series of binary classifiers arranged in the first order to second training results from training with the series of binary classifiers arranged in the second order.
13. The computer program product of claim 12, wherein the stored program instructions are stored in a computer readable storage device in a data processing system, and wherein the stored program instructions are transferred over a network from a remote data processing system.
14. The computer program product of claim 12, wherein the stored program instructions are stored in a computer readable storage device in a server data processing system, and wherein the stored program instructions are downloaded in response to a request over a network to a remote data processing system for use in a computer readable storage device associated with the remote data processing system, further comprising:
program instructions to meter use of the program instructions associated with the request; and program instructions to generate an invoice based on the metered use.
15. The computer program product of claim 12, wherein the generating of the training batch of data points comprises randomly selecting the data points from the training data.
16. The computer program product of claim 12, wherein each of the series of binary classifiers outputs a first value for data points predicted to be part of the class associated with the binary classifier and outputs a second value for data points predicted to be part of any other classes of the plurality of classes.
17. The computer program product of claim 12, wherein training of the multi-class classifier further comprises applying a multi-class voting classifier to any data points remaining unclassified after progressing through the series of binary classifiers.
18. A computer system comprising one or more processors and one or more computer readable storage media, and program instructions collectively stored on the one or more computer readable storage media, the program instructions executable by the one or more processors to cause the one or more processors to perform operations comprising:
generating a training batch of data points from training data associated with a plurality of classes; building a multi-class classifier having a series of binary classifiers arranged in a first order, wherein each of the series of binary classifiers is associated with a respective class of the plurality of classes; training the multi-class classifier with the series of binary classifiers being arranged in the first order using the training batch of data points, the training comprising, at each binary classifier in succession:
identifying data points as belonging to the associated class resulting in classified data points; and
updating the training batch to exclude the classified data points prior to the training batch being inputted into a subsequent binary classifier;
 modifying the multi-class classifier by changing the order of the series of binary classifiers resulting in the series of binary classifiers being arranged in a second order; repeating the training of the multi-class classifier with the series of binary classifiers being arranged in the second order using the training batch of data points; and selecting a final configuration of the multi-class classifier based at least in part on a comparison of firs
[Truncated to input limit]