Repository: openml/automlbenchmark (language: python)

Issue:
'NoResultError' raised by FLAML Framework During Benchmark Execution
Hello,

I have just installed AutoML Benchmarck in an environment running Python 3.9.17. However, when I execute the command `python runbenchmark.py flaml`, I encounter the following error:

NoResultError: `best_iteration` is only defined when early stopping is used.

Is there a workaround to resolve this issue?

Thank you.

[error_traceback.txt](https://github.com/openml/automlbenchmark/files/15469087/error_traceback.txt)



Discussion / hints:
I had the same problem; it was related to flaml version 1.2.4 being used. 
Force the latest version:
`python runbenchmark.py flaml:latest`
I'll look into what exactly is causing this issue, but in the meanwhile, I hope @israel-cj's fix works for you!
Thanks to both of you for the quick replies.

To enable execution with the tag "latest," I added the following entry to the file ./resources/frameworks_latest.yaml:

```
flaml:
  version: latest
```


However, `python runbenchmark.py flaml:latest` raised a different error.

[error_traceback_2.txt](https://github.com/openml/automlbenchmark/files/15470771/error_traceback_2.txt)

For me, it worked when I force (`-s force`) the setup mode as well (you only need to do this once):
`python runbenchmark.py flaml:latest -s force`

Fix patch:
diff --git a/frameworks/flaml/requirements.txt b/frameworks/flaml/requirements.txt
new file mode 100644
index 000000000..ea734de4a
--- /dev/null
+++ b/frameworks/flaml/requirements.txt
@@ -0,0 +1,1 @@
+pandas<3
\ No newline at end of file
diff --git a/frameworks/flaml/setup.sh b/frameworks/flaml/setup.sh
index 7fb003789..e81d4ba7f 100755
--- a/frameworks/flaml/setup.sh
+++ b/frameworks/flaml/setup.sh
@@ -7,25 +7,27 @@ PKG=${3:-"flaml"}
 
 . ${HERE}/../shared/setup.sh ${HERE} true
 
+OPTIONALS="[automl]"
 if [[ "$VERSION" == "latest" ]]; then
     VERSION="main"
-    OPTIONALS=""
 fi
 
 if [[ "$VERSION" == "benchmark" ]]; then
     VERSION="stable"
-    OPTIONALS="[catboost]"
+    OPTIONALS="[automl, catboost]"
 else
     PIP uninstall -y catboost
 fi
 
-
 if [[ "$VERSION" == "stable" ]]; then
-    PIP install --no-cache-dir -U "${PKG}${OPTIONALS}<2"
-elif [[ "$VERSION" =~ ^[0-9] ]]; then
+    PIP install --no-cache-dir -U "${PKG}${OPTIONALS}"
+elif [[ "$VERSION" =~ ^[0-1] ]]; then
+    PIP install --no-cache-dir -U ${PKG}${OPTIONALS}==${VERSION}
+    # FLAML 1.2.4 does not work with newer versions of xgboost
+    PIP install "xgboost<2"
+elif [[ "$VERSION" =~ ^[2-9] ]]; then
     PIP install --no-cache-dir -U ${PKG}${OPTIONALS}==${VERSION}
 else
-#    PIP install --no-cache-dir -e git+${REPO}@${VERSION}#egg=${PKG}
     TARGET_DIR="${HERE}/lib/${PKG}"
     rm -Rf ${TARGET_DIR}
     git clone --depth 1 --single-branch --branch ${VERSION} --recurse-submodules ${REPO} ${TARGET_DIR}


Test patch:
diff --git a/resources/frameworks_latest.yaml b/resources/frameworks_latest.yaml
index 44f924232..11e9fda38 100644
--- a/resources/frameworks_latest.yaml
+++ b/resources/frameworks_latest.yaml
@@ -37,6 +37,9 @@ autosklearn2:
 AutoWEKA:
   version: 'latest'
 
+flaml:
+  version: 'latest'
+
 GAMA:
   version: 'latest'