Few gives you up to BigMem - typically at least a gig. So if you have 16 
byte values you can support at least 50M

David

On 06/05/2012 03:43 PM, Victor Herrera wrote:
>> @@ -45,18 +44,61 @@ EXPORT Compare(DATASET(Types.DiscreteField) Dep,DATASET(l_result) Computed) := M
>>   	SHARED J := JOIN(Computed,Dep,LEFT.id=RIGHT.id AND LEFT.number=RIGHT.number,notediff(LEFT,RIGHT));
>>   	// Shows which classes were modeled as which classes
>>   	EXPORT Raw := TABLE(J,{classifier,c_actual,c_modeled,score,score_delta,sole_result,Cnt := COUNT(GROUP)},classifier,c_actual,c_modeled,score,score_delta,sole_result,MERGE);
>> -	EXPORT CrossAssignments := TABLE(J,{classifier,c_actual,c_modeled,Cnt := COUNT(GROUP)},classifier,c_actual,c_modeled,FEW);
>> -	EXPORT PrecisionByClass := TABLE(J,{classifier,c_actual, Precision := AVE(GROUP,IF(c_actual=c_modeled,100,0))},classifier,c_actual,FEW);
>> -	EXPORT HeadLine := TABLE(J,{classifier, Precision := AVE(GROUP,IF(c_actual=c_modeled,100,0))},classifier,FEW);
>> +	// Building the Confusion Matrix
>> +	SHARED ConfMatrix_Rec := RECORD
>> +		Types.t_FieldNumber classifier;	// The classifier in question (value of 'number' on outcome data)
>> +		Types.t_Discrete c_actual;			// The value of c provided
>> +		Types.t_Discrete c_modeled;			// The value produced by the classifier
>> +		Types.t_FieldNumber Cnt:=0;			// Number of occurences
>> +	END;
>> +	SHARED class_cnt := TABLE(Dep,{classifier:= number, c_actual:= value, Cnt:= COUNT(GROUP)},number, value); // Looking for class values
> Hi David,
>
> There are only few combinations of number-value; the purpose of this line
> is just to know the set of class values for each classifier (since it is
> possible to process more than one classifier at a time), most of
> classifiers are binary and the remaining have a small set of class values.
> Thus at the end the quantity of combinations of number-value really depends
> on how many classifiers are processed at a time, and using FEW give us a
> max number of 10,000 that I guess is enough i.e. it can process up to 5000
> binary classifiers.
>
> Regards
> Victor
>
>
>
> On Fri, Jun 1, 2012 at 6:14 PM, David Bayliss<
> reply@reply.github.com
>> wrote:
>>> @@ -45,18 +44,61 @@ EXPORT Compare(DATASET(Types.DiscreteField)
>> Dep,DATASET(l_result) Computed) := M
>>>        SHARED J := JOIN(Computed,Dep,LEFT.id=RIGHT.id AND
>> LEFT.number=RIGHT.number,notediff(LEFT,RIGHT));
>>>        // Shows which classes were modeled as which classes
>>>        EXPORT Raw :=
>> TABLE(J,{classifier,c_actual,c_modeled,score,score_delta,sole_result,Cnt :=
>> COUNT(GROUP)},classifier,c_actual,c_modeled,score,score_delta,sole_result,MERGE);
>>> -     EXPORT CrossAssignments :=
>> TABLE(J,{classifier,c_actual,c_modeled,Cnt :=
>> COUNT(GROUP)},classifier,c_actual,c_modeled,FEW);
>>> -     EXPORT PrecisionByClass := TABLE(J,{classifier,c_actual, Precision
>> := AVE(GROUP,IF(c_actual=c_modeled,100,0))},classifier,c_actual,FEW);
>>> -     EXPORT HeadLine := TABLE(J,{classifier, Precision :=
>> AVE(GROUP,IF(c_actual=c_modeled,100,0))},classifier,FEW);
>>> +     // Building the Confusion Matrix
>>> +     SHARED ConfMatrix_Rec := RECORD
>>> +             Types.t_FieldNumber classifier; // The classifier in
>> question (value of 'number' on outcome data)
>>> +             Types.t_Discrete c_actual;                      // The
>> value of c provided
>>> +             Types.t_Discrete c_modeled;                     // The
>> value produced by the classifier
>>> +             Types.t_FieldNumber Cnt:=0;                     // Number
>> of occurences
>>> +     END;
>>> +     SHARED class_cnt := TABLE(Dep,{classifier:= number, c_actual:=
>> value, Cnt:= COUNT(GROUP)},number, value); // Looking for class values
>>
>> How many combinations of number/value are we expecting - previously we
>> were using ,FEW which is much faster (if it works). Failing that using a
>> ,MERGE will prevent skew issues
>>
>> ---
>> Reply to this email directly or view it on GitHub:
>> https://github.com/hpcc-systems/ecl-ml/pull/120/files#r918004
>>
> ---
> Reply to this email directly or view it on GitHub:
> https://github.com/hpcc-systems/ecl-ml/pull/120/files#r931376