Repository: intel/ipex-llm
Pull request: Resolve inc metric duplicate registration
resolves https://github.com/intel-analytics/BigDL/issues/4002
In previous implementation, each call of quantization, a metric will be registered.  
This causes error when we call quantization with a same metric in one script twice.

### previous implementation :
Each metric is registered at runtime, so there can be name conflict.
```python
  self.metric = Metric(
      MyMetric,
      name="%s_%s" % (framework_metric.__name__, type(metric).__name__)
  )
```
### After this fix:
Customized metrics firstly are registered once for all statically.
```python
METRICS('pytorch').register('PytorchINCMetric', PytorchINCMetric)
METRICS('tensorflow').register('TensorflowINCMetric', TensorflowINCMetric)
```
Then at runtime, we modify the real metric inside `INCMetric` by:
```python
PytorchINCMetric.set_metric(metric)
```
And fetch the registered metric by class name:
```python
self.cfg.evaluation.accuracy.metric = {PytorchINCMetric.__name__: None}
```

### Modifications

- [x]  Add default criterion to INC, allowing 0.99 acc drop by default
- [x]  Fix metric registered in second time
- [x]  Add test with metric assigned second time


Problem statement:
Nano quantization report AssertionError on the second time 
Here is a minimal example:
```python
model = ResNet18(10, pretrained=False, include_top=False, freeze=True)
loss = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)
train_loader = create_data_loader(data_dir, batch_size, num_workers, data_transform)
trainer = Trainer(max_epochs=1)
pl_model = Trainer.compile(model, loss, optimizer)
qmodel = trainer.quantize(pl_model, train_loader, train_loader,
                              metric=torchmetrics.F1(10), framework='pytorch_fx',
                              approach='static',
                              tuning_strategy='basic',
                              accuracy_criterion={'relative': 0.99,
                                                  'higher_is_better': True})
qmodel = trainer.quantize(pl_model, train_loader, train_loader,
                              metric=torchmetrics.F1(10), framework='pytorch_fx',
                              approach='static',
                              tuning_strategy='basic',
                              accuracy_criterion={'relative': 0.99,
                                                  'higher_is_better': True})
```
it will report this on the second time.
```bash
Traceback (most recent call last):
  File "test.py", line 68, in <module>
    'higher_is_better': True})
  File "/home/junweid/BigDL/python/nano/src/bigdl/nano/pytorch/trainer/Trainer.py", line 239, in quantize
    metric)
  File "/home/junweid/BigDL/python/nano/src/bigdl/nano/quantization/neural_compressor/quantization.py", line 102, in post_training_quantize
    name="%s_%s" % (framework_metric.__name__, type(metric).__name__)
  File "/home/junweid/miniconda3/envs/chronos/lib/python3.7/site-packages/neural_compressor/experimental/quantization.py", line 287, in metric
    metrics.register(user_metric.name, user_metric.metric_cls)
  File "/home/junweid/miniconda3/envs/chronos/lib/python3.7/site-packages/neural_compressor/experimental/metric/metric.py", line 116, in register
    assert name not in self.metrics.keys(), 'registered metric name already exists.'
AssertionError: registered metric name already exists.
```

I am pretty sure this should be a INC issue, but we may have a look first. @zhentaocc

Hints from issue discussion:
It's not INC's problem. I think each call of `trainer.quantize(..)` will register the metric into INC as `F1`, so this is the reason.   
Maybe we should have better implementation in https://github.com/intel-analytics/BigDL/blob/branch-2.0/python/nano/src/bigdl/nano/quantization/neural_compressor/quantization.py#L90-L103.

Commit message: add extra test to make sure metric successfully assigned

Patch to review:
diff --git a/python/nano/src/bigdl/nano/quantization/neural_compressor/metric/base_metric.py b/python/nano/src/bigdl/nano/quantization/neural_compressor/metric/base_metric.py
index fa26ef7ab86..320311051cb 100644
--- a/python/nano/src/bigdl/nano/quantization/neural_compressor/metric/base_metric.py
+++ b/python/nano/src/bigdl/nano/quantization/neural_compressor/metric/base_metric.py
@@ -17,6 +17,8 @@
 
 
 class INCMetric(ABC):
+    METRIC_ID = 0
+
     def __init__(self):
         self.pred_list = []
         self.label_list = []
@@ -55,3 +57,8 @@ def stack(self, preds, labels):
     @abstractmethod
     def to_scalar(self, tensor):
         pass
+
+    @staticmethod
+    def get_next_metric_id():
+        INCMetric.METRIC_ID += 1
+        return INCMetric.METRIC_ID
diff --git a/python/nano/src/bigdl/nano/quantization/neural_compressor/quantization.py b/python/nano/src/bigdl/nano/quantization/neural_compressor/quantization.py
index fbeb737f4ab..aa0e8990b88 100644
--- a/python/nano/src/bigdl/nano/quantization/neural_compressor/quantization.py
+++ b/python/nano/src/bigdl/nano/quantization/neural_compressor/quantization.py
@@ -96,10 +96,9 @@ def __init__(self):
                         to the new local class to avoid that.
                         """
                         self.metric = metric
-
                 self.metric = Metric(
                     MyMetric,
-                    name="%s_%s" % (framework_metric.__name__, type(metric).__name__)
+                    name=f"nano_metric_{framework_metric.get_next_metric_id()}"
                 )
 
         quantized = self()
diff --git a/python/nano/test/pytorch/tests/test_trainer.py b/python/nano/test/pytorch/tests/test_trainer.py
index f018e27260a..360964a0126 100644
--- a/python/nano/test/pytorch/tests/test_trainer.py
+++ b/python/nano/test/pytorch/tests/test_trainer.py
@@ -82,11 +82,12 @@ def test_trainer_quantize_inc_ptq_compiled(self):
         train_loader_iter = iter(self.train_loader)
         trainer = Trainer(max_epochs=1)
         pl_model = Trainer.compile(self.model, self.loss, self.optimizer)
+        x = next(train_loader_iter)[0]
 
         # Case 1: Default
         qmodel = trainer.quantize(pl_model, self.train_loader)
         assert qmodel
-        out = qmodel(next(train_loader_iter)[0])
+        out = qmodel(x)
         assert out.shape == torch.Size([256, 10])
 
         # Case 2: Override by arguments
@@ -98,13 +99,13 @@ def test_trainer_quantize_inc_ptq_compiled(self):
                                                       'higher_is_better': True})
 
         assert qmodel
-        out = qmodel(next(train_loader_iter)[0])
+        out = qmodel(x)
         assert out.shape == torch.Size([256, 10])
 
         # Case 3: Dynamic quantization
         qmodel = trainer.quantize(pl_model, approach='dynamic')
         assert qmodel
-        out = qmodel(next(train_loader_iter)[0])
+        out = qmodel(x)
         assert out.shape == torch.Size([256, 10])
 
         # Case 4: Invalid approach
@@ -113,6 +114,13 @@ def test_trainer_quantize_inc_ptq_compiled(self):
                                              "{} is invalid.".format(invalid_approach)):
             trainer.quantize(pl_model, approach=invalid_approach)
 
+        # Case 5: Test if registered metric can be fetched successfully
+        qmodel = trainer.quantize(pl_model, self.train_loader, self.train_loader,
+                                  metric=torchmetrics.F1(10))
+        assert qmodel
+        out = qmodel(x)
+        assert out.shape == torch.Size([256, 10])
+
     def test_trainer_quantize_inc_ptq_customized(self):
         # Test if a Lightning Module not compiled by nano works
         train_loader_iter = iter(self.train_loader)


Code review comments:
diff_hunk: @@ -96,10 +96,9 @@ def __init__(self):
                         to the new local class to avoid that.
                         """
                         self.metric = metric
-
                 self.metric = Metric(
                     MyMetric,
-                    name="%s_%s" % (framework_metric.__name__, type(metric).__name__)
+                    name=f"nano_metric_{framework_metric.get_next_metric_id()}"
path: python/nano/src/bigdl/nano/quantization/neural_compressor/quantization.py
text: @user1:
give it a more meaningful name

@user2:
yeah, maybe a metric name and an unique id.

@author:
OK, let me roll back the name.