anon-iclr-submission/benchname-bug-localization huggingface dataset:
Repository: augerai/a2ml
Issue: ModuleNotFoundError: No module named 'arff'

Get 
ModuleNotFoundError: No module named 'arff'

on
a2ml import 

after
pip istall -e .



Changed files:
setup.py

diff --git a/setup.py b/setup.py
index 99fd53e..079996b 100644
--- a/setup.py
+++ b/setup.py
@@ -44,7 +44,8 @@ install_requires = [
     'smart_open==1.9.0',
     'jsonpickle',
     'websockets',
-    'HLL'
+    'HLL',
+    'liac-arff'    
 ]
 
 extras = {
@@ -68,8 +69,6 @@ extras = {
         'celery==4.4.0',
         'fastapi',
         'gevent',
-        'joblib',
-        'liac-arff',
         'redis',
         's3fs',
         'uvicorn',
Repository: augerai/a2ml
Issue: Store only actual and prediction_id on score_actuals task

malformed input like

```
"actual_records": [
      {
        "null": [
          "2.2",
          "5.0",
          "1.5",
          "Iris-virginica"
        ],
        "actual": "6.0",
        "prediction_id": "6efdf66d-40a2-4074-b3db-8900c622e07f"
      }
    ],
```
https://app-staging.auger.ai/admin/cluster_tasks/29100
https://app-staging.auger.ai/admin/experiment_sessions/5534

Is stored to feather:

```
,prediction_id,class,null,petal_length,petal_width,prediction_group_id,sepal_length,sepal_width
0,6efdf66d-40a2-4074-b3db-8900c622e07f,6.0,['2.2' '5.0' '1.5' 'Iris-virginica'],5.0,1.5,136eeb53-585f-4833-a3be-da4279f3aaa2,6.0,2.2
```



Changed files:
tests/model_review/test_model_review.py, a2ml/api/model_review/model_review.py

diff --git a/a2ml/api/model_review/model_review.py b/a2ml/api/model_review/model_review.py
index b468ae0..91d91d6 100644
--- a/a2ml/api/model_review/model_review.py
+++ b/a2ml/api/model_review/model_review.py
@@ -137,6 +137,8 @@ class ModelReview(object):
 
         ds_actuals = DataFrame.create_dataframe(actuals_path, actual_records,
             features=features)
+        if features is None:
+            ds_actuals.select(['prediction_id', 'actual'])
 
         actuals_count = ds_actuals.count()
 
diff --git a/tests/model_review/test_model_review.py b/tests/model_review/test_model_review.py
index 3e38995..184142c 100644
--- a/tests/model_review/test_model_review.py
+++ b/tests/model_review/test_model_review.py
@@ -393,8 +393,8 @@ def test_score_actuals_with_no_predictions_in_model_folder():
       os.remove(actuals_path)
 
     actuals = [
-      { 'prediction_id':'09aaa96b-5d9c-4c45-ab04-726da868624b', 'species':'versicolor' },
-      { 'prediction_id':'5e5ad22b-6789-47c6-9a4d-a3a998065127', 'species':'virginica' }
+      { 'prediction_id':'09aaa96b-5d9c-4c45-ab04-726da868624b', 'actual':'versicolor' },
+      { 'prediction_id':'5e5ad22b-6789-47c6-9a4d-a3a998065127', 'actual':'virginica' }
     ]
 
     try:
Repository: augerai/a2ml
Issue: DatasetValidationError on import_data_task in Azure

`DatasetValidationError` on `import_data_task` in Azure

https://app-staging.auger.ai/admin/cluster_tasks/81400 - this happened when I tried to load CSV file through UI, after I retried task it worked

https://app-staging.auger.ai/admin/cluster_tasks/83283 - this happened on review process, after retry it also worked

It happens time to time, Probably it's race conditions

cc @skatedplrn

Changed files:
a2ml/api/azure/dataset.py

diff --git a/a2ml/api/azure/dataset.py b/a2ml/api/azure/dataset.py
index 0d0c453..e84908c 100644
--- a/a2ml/api/azure/dataset.py
+++ b/a2ml/api/azure/dataset.py
@@ -7,6 +7,8 @@ from .exceptions import AzureException
 from a2ml.api.utils.decorators import error_handler, authenticated
 from .credentials import Credentials
 from  a2ml.api.utils.s3_fsclient import S3FSClient
+from a2ml.api.utils.dataframe import DataFrame
+
 
 class AzureDataset(object):
 
@@ -59,6 +61,14 @@ class AzureDataset(object):
                 if self.ctx.config.path and not local_path.startswith("/"):
                     local_path = os.path.join(self.ctx.config.path, local_path)
 
+                # Do it on Auger Cloud only
+                if fsclient.is_s3_path(source) and \\
+                    not (source.endswith(".csv") or source.endswith(".parquet")):
+                    self.ctx.log("Convert file %s to parquet format."%source)
+                    df = DataFrame.create_dataframe(local_path)
+                    df.saveToFile(local_path+".parquet")
+                    local_path = local_path+".parquet"
+
                 ds.upload_files(files=[local_path], relative_root=None,
                     target_path=None, overwrite=True, show_progress=True)
                 dataset_name = os.path.basename(local_path)