Repository: All-Hands-AI/OpenHands

Issue:
[Bug]: Runtime failed to build due to Mamba Error
### Is there an existing issue for the same bug?

- [X] I have checked the troubleshooting document at https://docs.all-hands.dev/modules/usage/troubleshooting
- [X] I have checked the existing issues.

### Describe the bug

```
Traceback (most recent call last):
  File "/openhands/miniforge3/bin/mamba", line 7, in <module>
    from mamba.mamba import main
  File "/openhands/miniforge3/lib/python3.11/site-packages/mamba/mamba.py", line 18, in <module>
    from conda.cli.main import generate_parser, init_loggers
ImportError: cannot import name 'generate_parser' from 'conda.cli.main' 
```

which cause the build to fail

### Current OpenHands version

```bash
I am experimenting on https://github.com/All-Hands-AI/OpenHands/pull/3985, but it should pretty much the same as main
```


### Installation and Configuration

```bash
make build
```


### Model and Agent

_No response_

### Operating System

_No response_

### Reproduction Steps

_No response_

### Logs, Errors, Screenshots, and Additional Context

```
...

  Package                         Version  Build               Channel         Size
─────────────────────────────────────────────────────────────────────────────────────
  Install:
─────────────────────────────────────────────────────────────────────────────────────

  + _libgcc_mutex                     0.1  conda_forge         conda-forge         
  + ca-certificates             2024.8.30  hbcca054_0          conda-forge         
  + ld_impl_linux-64                 2.40  hf3520f5_7          conda-forge         
  + pybind11-abi                        4  hd8ed1ab_3          conda-forge         
  + python_abi                       3.12  5_cp312             conda-forge         
  + tzdata                          2024a  h8827d51_1          conda-forge         
  + libgomp                        14.1.0  h77fa898_1          conda-forge         
  + _openmp_mutex                     4.5  2_gnu               conda-forge         
  + libgcc                         14.1.0  h77fa898_1          conda-forge         
  + libexpat                        2.6.3  h5888daf_0          conda-forge         
  + libgcc-ng                      14.1.0  h69a702a_1          conda-forge         
  + libstdcxx                      14.1.0  hc0a3c3a_1          conda-forge         
  + openssl                         3.3.2  hb9d3cd8_0          conda-forge         
  + bzip2                           1.0.8  h4bc722e_7          conda-forge         
  + c-ares                         1.32.3  h4bc722e_0          conda-forge         
  + keyutils                        1.6.1  h166bdaf_0          conda-forge         
  + libev                            4.33  hd590300_2          conda-forge         
  + libffi                          3.4.2  h7f98852_5          conda-forge         
  + libiconv                         1.17  hd590300_2          conda-forge         
  + libnsl                          2.0.1  hd590300_0          conda-forge         
  + libstdcxx-ng                   14.1.0  h4852527_1          conda-forge         
  + libuuid                        2.38.1  h0b41bf4_0          conda-forge         
  + libxcrypt                      4.4.36  hd590300_1          conda-forge         
  + libzlib                         1.3.1  h4ab18f5_1          conda-forge         
  + lzo                              2.10  hd590300_1001       conda-forge         
  + ncurses                           6.5  he02047a_1          conda-forge         
  + reproc                   14.2.4.post0  hd590300_1          conda-forge         
  + xz                              5.2.6  h166bdaf_0          conda-forge         
  + fmt                            10.2.1  h00ab1b0_0          conda-forge         
  + icu                              75.1  he02047a_0          conda-forge         
  + libedit                  3.1.20191231  he28a2e2_2          conda-forge         
  + libnghttp2                     1.58.0  h47da74e_1          conda-forge         
  + libsolv                        0.7.30  h3509ff9_0          conda-forge         
  + libsqlite                      3.46.1  hadc24fc_0          conda-forge         
  + libssh2                        1.11.0  h0841786_0          conda-forge         
  + lz4-c                           1.9.4  hcb278e6_0          conda-forge         
  + readline                          8.2  h8228510_1          conda-forge         
  + reproc-cpp               14.2.4.post0  h59595ed_1          conda-forge         
  + tk                             8.6.13  noxft_h4845f30_101  conda-forge         
  + yaml-cpp                        0.8.0  h59595ed_0          conda-forge         
  + zstd                            1.5.6  ha6fb4c9_0          conda-forge         
  + krb5                           1.21.3  h659f571_0          conda-forge         
  + libxml2                        2.12.7  he7c6b58_4          conda-forge         
  + python                         3.12.6  hc5c86c4_1_cpython  conda-forge         
  + libarchive                      3.7.4  hfca40fe_0          conda-forge         
  + libcurl                        8.10.1  hbbe4b11_0          conda-forge         
  + menuinst                        2.1.2  py312h7900ff3_1     conda-forge         
  + archspec                        0.2.3  pyhd8ed1ab_0        conda-forge         
  + boltons                        24.0.0  pyhd8ed1ab_0        conda-forge         
  + brotli-python                   1.1.0  py312h2ec8cdc_2     conda-forge         
  + certifi                     2024.8.30  pyhd8ed1ab_0        conda-forge         
  + charset-normalizer              3.3.2  pyhd8ed1ab_0        conda-forge         
  + colorama                        0.4.6  pyhd8ed1ab_0        conda-forge         
  + distro                          1.9.0  pyhd8ed1ab_0        conda-forge         
  + frozendict                      2.4.4  py312h66e93f0_1     conda-forge         
  + hpack                           4.0.0  pyh9f0ad1d_0        conda-forge         
  + hyperframe                      6.0.1  pyhd8ed1ab_0        conda-forge         
  + idna                             3.10  pyhd8ed1ab_0        conda-forge         
  + jsonpointer                     3.0.0  py312h7900ff3_1     conda-forge         
  + libmamba                        1.5.9  h4cc3d14_0          conda-forge         
  + packaging                        24.1  pyhd8ed1ab_0        conda-forge         
  + platformdirs                    4.3.6  pyhd8ed1ab_0        conda-forge         
  + pluggy                          1.5.0  pyhd8ed1ab_0        conda-forge         
  + pycosat                         0.6.6  py312h98912ed_0     conda-forge         
  + pycparser                        2.22  pyhd8ed1ab_0        conda-forge         
  + pysocks                         1.7.1  pyha2e5f31_6        conda-forge         
  + ruamel.yaml.clib                0.2.8  py312h98912ed_0     conda-forge         
  + setuptools                     74.1.2  pyhd8ed1ab_0        conda-forge         
  + truststore                      0.9.2  pyhd8ed1ab_0        conda-forge         
  + wheel                          0.44.0  pyhd8ed1ab_0        conda-forge         
  + cffi                           1.17.1  py312h06ac9bb_0     conda-forge         
  + h2                              4.1.0  pyhd8ed1ab_0        conda-forge         
  + jsonpatch                        1.33  pyhd8ed1ab_0        conda-forge         
  + libmambapy                      1.5.9  py312h7fb9e8e_0     conda-forge         
  + pip                              24.2  pyh8b19718_1        conda-forge         
  + ruamel.yaml                    0.18.6  py312h98912ed_0     conda-forge         
  + tqdm                           4.66.5  pyhd8ed1ab_0        conda-forge         
  + zstandard                      0.23.0  py312hef9b889_1     conda-forge         
  + conda-package-streaming        0.10.0  pyhd8ed1ab_0        conda-forge         
  + urllib3                         2.2.3  pyhd8ed1ab_0        conda-forge         
  + conda-package-handling          2.3.0  pyh7900ff3_0        conda-forge         
  + requests                       2.32.3  pyhd8ed1ab_0        conda-forge         
  + conda                          24.7.1  py312h7900ff3_0     conda-forge         
  + conda-libmamba-solver          24.7.0  pyhd8ed1ab_0        conda-forge         
  + mamba                           1.5.9  py312h9460a1c_0     conda-forge         

  Summary:

  Install: 85 packages

  Total download: 0 B

─────────────────────────────────────────────────────────────────────────────────────



Transaction starting

Transaction finished

To activate this environment, use:

    micromamba activate /openhands/miniforge3

Or to execute a single command in this environment, use:

    micromamba run -p /openhands/miniforge3 mycommand

installation finished.
Warning: 'conda-forge' already in 'channels' list, moving to the bottom
Removing intermediate container 8a5d0a595e2d
 ---> 6b53437dc1db
Step 7/11 : RUN /openhands/miniforge3/bin/mamba install conda-forge::poetry python=3.11 -y
 ---> Running in 71c23bfe4fd5
Transaction

  Prefix: /openhands/miniforge3

  Updating specs:

   - conda-forge::poetry
   - python=3.11
   - ca-certificates
   - certifi
   - openssl


  Package                           Version  Build               Channel           Size
─────────────────────────────────────────────────────────────────────────────────────────
  Install:
─────────────────────────────────────────────────────────────────────────────────────────

  + expat                             2.6.3  h6a678d5_0          pkgs/main        181kB
  + blas                                1.0  openblas            pkgs/main         47kB
  + libgfortran5                     11.2.0  h1234567_1          pkgs/main          2MB
  + libgfortran-ng                   11.2.0  h00389a5_1          pkgs/main         20kB
  + libopenblas                      0.3.21  h043d6bf_0          pkgs/main          6MB
  + zlib                             1.2.13  h4ab18f5_6          conda-forge       93kB
  + pcre2                             10.42  hebb0a14_1          pkgs/main          1MB
  + sqlite                           3.45.3  h5eee18b_0          pkgs/main          1MB
  + libglib                          2.78.4  hdc74915_0          pkgs/main          2MB
  + glib-tools                       2.78.4  h6a678d5_0          pkgs/main        118kB
  + python-fastjsonschema            2.20.0  pyhd8ed1ab_0        conda-forge      226kB
  + poetry-core                       1.9.0  pyhd8ed1ab_0        conda-forge      227kB
  + tomlkit                          0.13.2  pyha770c72_0        conda-forge       37kB
  + numpy-base                       1.26.4  py311hbfb1bba_0     pkgs/main          9MB
  + more-itertools                   10.3.0  py311h06a4308_0     pkgs/main        149kB
  + msgpack-python                    1.0.3  py311hdb19cb5_0     pkgs/main         37kB
  + glib                             2.78.4  h6a678d5_0          pkgs/main        520kB
  + zipp                             3.17.0  py311h06a4308_0     pkgs/main         25kB
  + filelock                         3.13.1  py311h06a4308_0     pkgs/main         25kB
  + distlib                           0.3.8  py311h06a4308_0     pkgs/main        467kB
  + trove-classifiers            2023.10.18  py311h06a4308_0     pkgs/main         22kB
  + tomli                             2.0.1  py311h06a4308_0     pkgs/main         31kB
  + shellingham                       1.5.0  py311h06a4308_0     pkgs/main         21kB
  + pyproject_hooks                   1.0.0  py311h06a4308_0     pkgs/main         25kB
  + pkginfo                          1.10.0  py311h06a4308_0     pkgs/main         67kB
  + crashtest                         0.4.1  py311h06a4308_0     pkgs/main         18kB
  + numpy                            1.26.4  py311h24aa872_0     pkgs/main         11kB
  + dbus                            1.13.18  hb2f20db_0          pkgs/main        516kB
  + importlib-metadata                7.0.1  py311h06a4308_0     pkgs/main         50kB
  + virtualenv                      20.26.1  py311h06a4308_0     pkgs/main          4MB
  + cryptography                     43.0.0  py311hdda0065_0     pkgs/main          2MB
  + rapidfuzz                         3.5.2  py311h6a678d5_0     pkgs/main          2MB
  + cleo                              2.1.0  py311h06a4308_0     pkgs/main        181kB
  + jeepney                           0.7.1  pyhd3eb1b0_0        pkgs/main         39kB
  + ptyprocess                        0.7.0  pyhd3eb1b0_2        pkgs/main         17kB
  + python-installer                  0.7.0  pyhd3eb1b0_1        pkgs/main        257kB
  + jaraco.classes                    3.2.1  pyhd3eb1b0_0        pkgs/main          9kB
  + importlib_metadata                7.0.1  hd3eb1b0_0          pkgs/main          9kB
  + pexpect                           4.8.0  pyhd3eb1b0_3        pkgs/main         54kB
  + python-build                      1.2.2  pyhd8ed1ab_0        conda-forge       25kB
  + secretstorage                     3.3.1  py311h06a4308_1     pkgs/main         29kB
  + dulwich                          0.21.3  py311h5eee18b_0     pkgs/main        998kB
  + requests-toolbelt                 1.0.0  py311h06a4308_0     pkgs/main         92kB
  + cachecontrol                     0.14.0  py311h06a4308_1     pkgs/main         46kB
  + keyring                          24.3.1  py311h06a4308_0     pkgs/main         82kB
  + cachecontrol-with-filecache      0.14.0  py311h06a4308_1     pkgs/main          5kB
  + poetry                            1.8.3  linux_pyha804496_1  conda-forge      167kB
  + poetry-plugin-export              1.8.0  pyhd8ed1ab_0        conda-forge       16kB

  Change:
─────────────────────────────────────────────────────────────────────────────────────────

  - ruamel.yaml.clib                  0.2.8  py312h98912ed_0     conda-forge     Cached
  + ruamel.yaml.clib                  0.2.8  py311h5eee18b_0     pkgs/main        161kB
  - pycosat                           0.6.6  py312h98912ed_0     conda-forge     Cached
  + pycosat                           0.6.6  py311h5eee18b_1     pkgs/main         97kB
  - menuinst                          2.1.2  py312h7900ff3_1     conda-forge     Cached
  + menuinst                          2.1.2  py311h06a4308_0     pkgs/main        263kB
  - cffi                             1.17.1  py312h06ac9bb_0     conda-forge     Cached
  + cffi                             1.17.1  py311h1fdaa30_0     pkgs/main        322kB

  Reinstall:
─────────────────────────────────────────────────────────────────────────────────────────

  o wheel                            0.44.0  pyhd8ed1ab_0        conda-forge     Cached
  o setuptools                       74.1.2  pyhd8ed1ab_0        conda-forge     Cached
  o pip                                24.2  pyh8b19718_1        conda-forge     Cached
  o truststore                        0.9.2  pyhd8ed1ab_0        conda-forge     Cached
  o pysocks                           1.7.1  pyha2e5f31_6        conda-forge     Cached
  o pycparser                          2.22  pyhd8ed1ab_0        conda-forge     Cached
  o pluggy                            1.5.0  pyhd8ed1ab_0        conda-forge     Cached
  o platformdirs                      4.3.6  pyhd8ed1ab_0        conda-forge     Cached
  o packaging                          24.1  pyhd8ed1ab_0        conda-forge     Cached
  o idna                               3.10  pyhd8ed1ab_0        conda-forge     Cached
  o hyperframe                        6.0.1  pyhd8ed1ab_0        conda-forge     Cached
  o hpack                             4.0.0  pyh9f0ad1d_0        conda-forge     Cached
  o distro                            1.9.0  pyhd8ed1ab_0        conda-forge     Cached
  o colorama                          0.4.6  pyhd8ed1ab_0        conda-forge     Cached
  o charset-normalizer                3.3.2  pyhd8ed1ab_0        conda-forge     Cached
  o certifi                       2024.8.30  pyhd8ed1ab_0        conda-forge     Cached
  o boltons                          24.0.0  pyhd8ed1ab_0        conda-forge     Cached
  o archspec                          0.2.3  pyhd8ed1ab_0        conda-forge     Cached
  o h2                                4.1.0  pyhd8ed1ab_0        conda-forge     Cached
  o tqdm                             4.66.5  pyhd8ed1ab_0        conda-forge     Cached
  o urllib3                           2.2.3  pyhd8ed1ab_0        conda-forge     Cached
  o conda-package-streaming          0.10.0  pyhd8ed1ab_0        conda-forge     Cached
  o jsonpatch                          1.33  pyhd8ed1ab_0        conda-forge     Cached
  o requests                         2.32.3  pyhd8ed1ab_0        conda-forge     Cached
  o conda-package-handling            2.3.0  pyh7900ff3_0        conda-forge     Cached
  o conda-libmamba-solver            24.7.0  pyhd8ed1ab_0        conda-forge     Cached

  Upgrade:
─────────────────────────────────────────────────────────────────────────────────────────

  - ca-certificates               2024.8.30  hbcca054_0          conda-forge     Cached
  + ca-certificates               2024.9.24  h06a4308_0          pkgs/main        133kB
  - xz                                5.2.6  h166bdaf_0          conda-forge     Cached
  + xz                                5.4.6  h5eee18b_1          pkgs/main        659kB
  - libxml2                          2.12.7  he7c6b58_4          conda-forge     Cached
  + libxml2                          2.13.1  hfdd30dd_2          pkgs/main        757kB
  - conda                            24.7.1  py312h7900ff3_0     conda-forge     Cached
  + conda                            24.9.0  py311h06a4308_0     pkgs/main          1MB

  Downgrade:
─────────────────────────────────────────────────────────────────────────────────────────

  - libuuid                          2.38.1  h0b41bf4_0          conda-forge     Cached
  + libuuid                          1.41.5  h5eee18b_0          pkgs/main         28kB
  - icu                                75.1  he02047a_0          conda-forge     Cached
  + icu                                73.1  h6a678d5_0          pkgs/main         27MB
  - libzlib                           1.3.1  h4ab18f5_1          conda-forge     Cached
  + libzlib                          1.2.13  h4ab18f5_6          conda-forge       62kB
  - libsqlite                        3.46.1  hadc24fc_0          conda-forge     Cached
  + libsqlite                        3.46.0  hde9e2c9_0          conda-forge      865kB
  - libcurl                          8.10.1  hbbe4b11_0          conda-forge     Cached
  + libcurl                           8.8.0  hca28451_1          conda-forge      410kB
  - libsolv                          0.7.30  h3509ff9_0          conda-forge     Cached
  + libsolv                          0.7.24  he621ea3_1          pkgs/main        502kB
  - python                           3.12.6  hc5c86c4_1_cpython  conda-forge     Cached
  + python                           3.11.9  h955ad1f_0          pkgs/main         34MB
  - libmamba                          1.5.9  h4cc3d14_0          conda-forge     Cached
  + libmamba                          1.5.8  had39da4_0          conda-forge        2MB
  - frozendict                        2.4.4  py312h66e93f0_1     conda-forge     Cached
  + frozendict                        2.4.2  py311h06a4308_0     pkgs/main         38kB
  - brotli-python                     1.1.0  py312h2ec8cdc_2     conda-forge     Cached
  + brotli-python                     1.0.9  py311h6a678d5_8     pkgs/main        367kB
  - ruamel.yaml                      0.18.6  py312h98912ed_0     conda-forge     Cached
  + ruamel.yaml                     0.17.21  py311h5eee18b_0     pkgs/main        255kB
  - zstandard                        0.23.0  py312hef9b889_1     conda-forge     Cached
  + zstandard                        0.19.0  py311h5eee18b_0     pkgs/main        449kB
  - jsonpointer                       3.0.0  py312h7900ff3_1     conda-forge     Cached
  + jsonpointer                         2.1  pyhd3eb1b0_0        pkgs/main          9kB
  - python_abi                         3.12  5_cp312             conda-forge     Cached
  + python_abi                         3.11  2_cp311             conda-forge        5kB
  - libmambapy                        1.5.9  py312h7fb9e8e_0     conda-forge     Cached
  + libmambapy                        1.5.8  py311hf2555c7_0     conda-forge      311kB
  - mamba                             1.5.9  py312h9460a1c_0     conda-forge     Cached
  + mamba                             1.5.8  py311h3072747_0     conda-forge       67kB

  Summary:

  Install: 48 packages
  Change: 4 packages
  Reinstall: 26 packages
  Upgrade: 4 packages
  Downgrade: 16 packages

  Total download: 104MB

─────────────────────────────────────────────────────────────────────────────────────────



Looking for: ['conda-forge::poetry', 'python=3.11']


Downloading and Extracting Packages: ...working... done
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
Removing intermediate container 71c23bfe4fd5
 ---> e07c93afbda8
Step 8/11 : RUN if [ -d /openhands/code ]; then rm -rf /openhands/code; fi
 ---> Running in 13bd633e1905
Removing intermediate container 13bd633e1905
 ---> 172894098a49
Step 9/11 : COPY ./code /openhands/code
 ---> 7d7e9aa1e383
Step 10/11 : WORKDIR /openhands/code
 ---> Running in 35bca73c59ee
Removing intermediate container 35bca73c59ee
 ---> ce2e67b74866
Step 11/11 : RUN     /openhands/miniforge3/bin/mamba run -n base poetry config virtualenvs.path /openhands/poetry &&     /openhands/miniforge3/bin/mamba run -n base poetry env use python3.11 &&     /openhands/miniforge3/bin/mamba run -n base poetry install --only main,runtime --no-interaction --no-root &&     apt-get update &&     /openhands/miniforge3/bin/mamba run -n base poetry run pip install playwright &&     /openhands/miniforge3/bin/mamba run -n base poetry run playwright install --with-deps chromium &&     echo "OH_INTERPRETER_PATH=$(/openhands/miniforge3/bin/mamba run -n base poetry run python -c "import sys; print(sys.executable)")" >> /etc/environment &&           /openhands/miniforge3/bin/mamba run -n base poetry cache clear --all . &&     chmod -R g+rws /openhands/poetry &&      mkdir -p /openhands/workspace && chmod -R g+rws,o+rw /openhands/workspace &&     apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* &&     /openhands/miniforge3/bin/mamba clean --all
 ---> Running in 0b64bd15c323
Traceback (most recent call last):
  File "/openhands/miniforge3/bin/mamba", line 7, in <module>
    from mamba.mamba import main
  File "/openhands/miniforge3/lib/python3.11/site-packages/mamba/mamba.py", line 18, in <module>
    from conda.cli.main import generate_parser, init_loggers
ImportError: cannot import name 'generate_parser' from 'conda.cli.main' (/openhands/miniforge3/lib/python3.11/site-packages/conda/cli/main.py)
The command '/bin/sh -c /openhands/miniforge3/bin/mamba run -n base poetry config virtualenvs.path /openhands/poetry &&     /openhands/miniforge3/bin/mamba run -n base poetry env use python3.11 &&     /openhands/miniforge3/bin/mamba run -n base poetry install --only main,runtime --no-interaction --no-root &&     apt-get update &&     /openhands/miniforge3/bin/mamba run -n base poetry run pip install playwright &&     /openhands/miniforge3/bin/mamba run -n base poetry run playwright install --with-deps chromium &&     echo "OH_INTERPRETER_PATH=$(/openhands/miniforge3/bin/mamba run -n base poetry run python -c "import sys; print(sys.executable)")" >> /etc/environment &&           /openhands/miniforge3/bin/mamba run -n base poetry cache clear --all . &&     chmod -R g+rws /openhands/poetry &&      mkdir -p /openhands/workspace && chmod -R g+rws,o+rw /openhands/workspace &&     apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* &&     /openhands/miniforge3/bin/mamba clean --all' returned a non-zero code: 1
```

Discussion:
tobitege: in the docker script:
```
RUN mkdir -p /openhands/micromamba &&     curl -Ls https://micro.mamba.pm/api/micromamba/linux-64/latest | tar -xvj -C /openhands/micromamba
```
I think the `tar` is creating a different folder structure `below` micromamba? 
Maybe this works?
```
RUN curl -Ls https://micro.mamba.pm/api/micromamba/linux-64/latest | tar -xvj -C /tmp && \
    mv /tmp/micromamba /openhands/

# Now the path will be correct
RUN /openhands/micromamba/bin/micromamba create -n openhands python=3.11 -y
RUN /openhands/micromamba/bin/micromamba install -n openhands -c conda-forge poetry -y
```
xingyaoww: You are probably right! Though the weird thing is i can build the exact dockerfile locally, as well as on the remote runtime 😓. I confirmed that this PR allow the previously fail image to successfully build and run
enyst: Just to note for the record: when we have switched from `miniconda` to `miniforge3`, part of the discussion there had a licensing reason, because anaconda's own licensing is messy. I made a quick check around `micromamba` from the licensing point of view and it looks good to me. 👍 

(it's similar to `miniforge3`, BSD 3-clause licensed software, scripts, default `conda-forge` channel.)
xingyaoww: Set CONDA_FORGE_YES to yes seems to also include Anaconda's commercial source when I'm testing the RemoteRuntime:
```
 ---> be5325871eaf
Step 8/12 : RUN /openhands/micromamba/bin/micromamba create -n openhands python=3.11 -y &&     /openhands/micromamba/bin/micromamba install -n openhands conda-forge::poetry -y
 ---> Running in ee1c0d20abcf
warning  libmamba 'repo.anaconda.com', a commercial channel hosted by Anaconda.com, is used.
    
warning  libmamba Please make sure you understand Anaconda Terms of Services.
    
warning  libmamba See: https://legal.anaconda.com/policies/en/
warning  libmamba 'repo.anaconda.com', a commercial channel hosted by Anaconda.com, is used.
    
warning  libmamba Please make sure you understand Anaconda Terms of Services.
    
warning  libmamba See: https://legal.anaconda.com/policies/en/
warning  libmamba 'repo.anaconda.com', a commercial channel hosted by Anaconda.com, is used.
    
warning  libmamba Please make sure you understand Anaconda Terms of Services.
    
warning  libmamba See: https://legal.anaconda.com/policies/en/
warning  libmamba 'repo.anaconda.com', a commercial channel hosted by Anaconda.com, is used.
    
warning  libmamba Please make sure you understand Anaconda Terms of Services.
    
warning  libmamba See: https://legal.anaconda.com/policies/en/

error    libmamba Could not lock non-existing path '/root/.mamba/pkgs'
```

Try set it back to no to see if it work (i think `conda-forge::poetry` which be equivalent to `-c conda-forge` which might work?)
enyst: > Set CONDA_FORGE_YES to yes seems to also include Anaconda's commercial source when I'm testing the RemoteRuntime:

That's strange! I think [the install script](https://raw.githubusercontent.com/mamba-org/micromamba-releases/main/install.sh) does this?
```
# Initializing conda-forge
case "$CONDA_FORGE_YES" in
  y|Y|yes)
    "${BIN_FOLDER}/micromamba" config append channels conda-forge
    "${BIN_FOLDER}/micromamba" config append channels nodefaults
    "${BIN_FOLDER}/micromamba" config set channel_priority strict
    ;;
esac
```


Patch:
diff --git a/.github/workflows/ghcr-build.yml b/.github/workflows/ghcr-build.yml
index 34824777..82c30d98 100644
--- a/.github/workflows/ghcr-build.yml
+++ b/.github/workflows/ghcr-build.yml
@@ -293,7 +293,7 @@ jobs:
           SANDBOX_RUNTIME_CONTAINER_IMAGE=$image_name \
           TEST_IN_CI=true \
           RUN_AS_OPENHANDS=false \
-          poetry run pytest -n 3 -raR --reruns 1 --reruns-delay 3 --cov=agenthub --cov=openhands --cov-report=xml -s ./tests/runtime
+          poetry run pytest -n 3 -raRs --reruns 2 --reruns-delay 5 --cov=agenthub --cov=openhands --cov-report=xml -s ./tests/runtime
       - name: Upload coverage to Codecov
         uses: codecov/codecov-action@v4
         env:
@@ -371,7 +371,7 @@ jobs:
           SANDBOX_RUNTIME_CONTAINER_IMAGE=$image_name \
           TEST_IN_CI=true \
           RUN_AS_OPENHANDS=true \
-          poetry run pytest -n 3 -raR --reruns 1 --reruns-delay 3 --cov=agenthub --cov=openhands --cov-report=xml -s ./tests/runtime
+          poetry run pytest -n 3 -raRs --reruns 2 --reruns-delay 5 --cov=agenthub --cov=openhands --cov-report=xml -s ./tests/runtime
       - name: Upload coverage to Codecov
         uses: codecov/codecov-action@v4
         env:
diff --git a/openhands/core/config/sandbox_config.py b/openhands/core/config/sandbox_config.py
index e6dc72d2..3f535a5f 100644
--- a/openhands/core/config/sandbox_config.py
+++ b/openhands/core/config/sandbox_config.py
@@ -18,6 +18,7 @@ class SandboxConfig:
         enable_auto_lint: Whether to enable auto-lint.
         use_host_network: Whether to use the host network.
         initialize_plugins: Whether to initialize plugins.
+        force_rebuild_runtime: Whether to force rebuild the runtime image.
         runtime_extra_deps: The extra dependencies to install in the runtime image (typically used for evaluation).
             This will be rendered into the end of the Dockerfile that builds the runtime image.
             It can contain any valid shell commands (e.g., pip install numpy).
@@ -43,6 +44,7 @@ class SandboxConfig:
     )
     use_host_network: bool = False
     initialize_plugins: bool = True
+    force_rebuild_runtime: bool = False
     runtime_extra_deps: str | None = None
     runtime_startup_env_vars: dict[str, str] = field(default_factory=dict)
     browsergym_eval_env: str | None = None
diff --git a/openhands/runtime/builder/docker.py b/openhands/runtime/builder/docker.py
index 72995510..56a759df 100644
--- a/openhands/runtime/builder/docker.py
+++ b/openhands/runtime/builder/docker.py
@@ -113,8 +113,8 @@ class DockerRuntimeBuilder(RuntimeBuilder):
                 raise subprocess.CalledProcessError(
                     return_code,
                     process.args,
-                    output=None,
-                    stderr=None,
+                    output=process.stdout.read() if process.stdout else None,
+                    stderr=process.stderr.read() if process.stderr else None,
                 )
 
         except subprocess.CalledProcessError as e:
diff --git a/openhands/runtime/client/runtime.py b/openhands/runtime/client/runtime.py
index c665f668..c6bb3019 100644
--- a/openhands/runtime/client/runtime.py
+++ b/openhands/runtime/client/runtime.py
@@ -167,6 +167,7 @@ class EventStreamRuntime(Runtime):
                 self.base_container_image,
                 self.runtime_builder,
                 extra_deps=self.config.sandbox.runtime_extra_deps,
+                force_rebuild=self.config.sandbox.force_rebuild_runtime,
             )
         self.container = self._init_container(
             sandbox_workspace_dir=self.config.workspace_mount_path_in_sandbox,  # e.g. /workspace
@@ -273,7 +274,7 @@ class EventStreamRuntime(Runtime):
             container = self.docker_client.containers.run(
                 self.runtime_container_image,
                 command=(
-                    f'/openhands/miniforge3/bin/mamba run --no-capture-output -n base '
+                    f'/openhands/micromamba/bin/micromamba run -n openhands '
                     f'poetry run '
                     f'python -u -m openhands.runtime.client.client {self._container_port} '
                     f'--working-dir "{sandbox_workspace_dir}" '
diff --git a/openhands/runtime/plugins/jupyter/__init__.py b/openhands/runtime/plugins/jupyter/__init__.py
index b46714c2..48ee21db 100644
--- a/openhands/runtime/plugins/jupyter/__init__.py
+++ b/openhands/runtime/plugins/jupyter/__init__.py
@@ -28,7 +28,8 @@ class JupyterPlugin(Plugin):
                 'cd /openhands/code\n'
                 'export POETRY_VIRTUALENVS_PATH=/openhands/poetry;\n'
                 'export PYTHONPATH=/openhands/code:$PYTHONPATH;\n'
-                '/openhands/miniforge3/bin/mamba run -n base '
+                'export MAMBA_ROOT_PREFIX=/openhands/micromamba;\n'
+                '/openhands/micromamba/bin/micromamba run -n openhands '
                 'poetry run jupyter kernelgateway '
                 '--KernelGatewayApp.ip=0.0.0.0 '
                 f'--KernelGatewayApp.port={self.kernel_gateway_port}\n'
diff --git a/openhands/runtime/remote/runtime.py b/openhands/runtime/remote/runtime.py
index c121021e..be4a19bc 100644
--- a/openhands/runtime/remote/runtime.py
+++ b/openhands/runtime/remote/runtime.py
@@ -119,6 +119,7 @@ class RemoteRuntime(Runtime):
                 self.config.sandbox.base_container_image,
                 self.runtime_builder,
                 extra_deps=self.config.sandbox.runtime_extra_deps,
+                force_rebuild=self.config.sandbox.force_rebuild_runtime,
             )
 
             response = send_request(
@@ -144,8 +145,8 @@ class RemoteRuntime(Runtime):
         start_request = {
             'image': self.container_image,
             'command': (
-                f'/openhands/miniforge3/bin/mamba run --no-capture-output -n base '
-                'PYTHONUNBUFFERED=1 poetry run '
+                f'/openhands/micromamba/bin/micromamba run -n openhands '
+                'poetry run '
                 f'python -u -m openhands.runtime.client.client {self.port} '
                 f'--working-dir {self.config.workspace_mount_path_in_sandbox} '
                 f'{plugin_arg}'
diff --git a/openhands/runtime/utils/runtime_templates/Dockerfile.j2 b/openhands/runtime/utils/runtime_templates/Dockerfile.j2
index 57ea62db..33958471 100644
--- a/openhands/runtime/utils/runtime_templates/Dockerfile.j2
+++ b/openhands/runtime/utils/runtime_templates/Dockerfile.j2
@@ -1,11 +1,13 @@
-{% if skip_init %}
 FROM {{ base_image }}
-{% else %}
+
+# Shared environment variables (regardless of init or not)
+ENV POETRY_VIRTUALENVS_PATH=/openhands/poetry
+ENV MAMBA_ROOT_PREFIX=/openhands/micromamba
+
+{% if not skip_init %}
 # ================================================================
 # START: Build Runtime Image from Scratch
 # ================================================================
-FROM {{ base_image }}
-
 {% if 'ubuntu' in base_image and (base_image.endswith(':latest') or base_image.endswith(':24.04')) %}
 {% set LIBGL_MESA = 'libgl1' %}
 {% else %}
@@ -14,7 +16,7 @@ FROM {{ base_image }}
 
 # Install necessary packages and clean up in one layer
 RUN apt-get update && \
-    apt-get install -y wget sudo apt-utils {{ LIBGL_MESA }} libasound2-plugins git && \
+    apt-get install -y wget curl sudo apt-utils {{ LIBGL_MESA }} libasound2-plugins git && \
     apt-get clean && \
     rm -rf /var/lib/apt/lists/*
 
@@ -26,19 +28,16 @@ RUN mkdir -p /openhands && \
     mkdir -p /openhands/logs && \
     mkdir -p /openhands/poetry
 
-# Directory containing subdirectories for virtual environment.
-ENV POETRY_VIRTUALENVS_PATH=/openhands/poetry
+# Install micromamba
+RUN mkdir -p /openhands/micromamba/bin && \
+    /bin/bash -c "PREFIX_LOCATION=/openhands/micromamba BIN_FOLDER=/openhands/micromamba/bin INIT_YES=no CONDA_FORGE_YES=yes $(curl -L https://micro.mamba.pm/install.sh)" && \
+    /openhands/micromamba/bin/micromamba config remove channels defaults && \
+    /openhands/micromamba/bin/micromamba config list
 
-RUN if [ ! -d /openhands/miniforge3 ]; then \
-    wget --progress=bar:force -O Miniforge3.sh "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh" && \
-    bash Miniforge3.sh -b -p /openhands/miniforge3 && \
-    rm Miniforge3.sh && \
-    chmod -R g+w /openhands/miniforge3 && \
-    bash -c ". /openhands/miniforge3/etc/profile.d/conda.sh && conda config --set changeps1 False && conda config --append channels conda-forge"; \
-    fi
+# Create the openhands virtual environment and install poetry and python
+RUN /openhands/micromamba/bin/micromamba create -n openhands -y && \
+    /openhands/micromamba/bin/micromamba install -n openhands -c conda-forge poetry python=3.11 -y
 
-# Install Python and Poetry
-RUN /openhands/miniforge3/bin/mamba install conda-forge::poetry python=3.11 -y
 # ================================================================
 # END: Build Runtime Image from Scratch
 # ================================================================
@@ -59,27 +58,28 @@ COPY ./code /openhands/code
 # virtual environment are used by default.
 WORKDIR /openhands/code
 RUN \
+    /openhands/micromamba/bin/micromamba config set changeps1 False && \
     # Configure Poetry and create virtual environment
-    /openhands/miniforge3/bin/mamba run -n base poetry config virtualenvs.path /openhands/poetry && \
-    /openhands/miniforge3/bin/mamba run -n base poetry env use python3.11 && \
+    /openhands/micromamba/bin/micromamba run -n openhands poetry config virtualenvs.path /openhands/poetry && \
+    /openhands/micromamba/bin/micromamba run -n openhands poetry env use python3.11 && \
     # Install project dependencies
-    /openhands/miniforge3/bin/mamba run -n base poetry install --only main,runtime --no-interaction --no-root && \
+    /openhands/micromamba/bin/micromamba run -n openhands poetry install --only main,runtime --no-interaction --no-root && \
     # Update and install additional tools
     apt-get update && \
-    /openhands/miniforge3/bin/mamba run -n base poetry run pip install playwright && \
-    /openhands/miniforge3/bin/mamba run -n base poetry run playwright install --with-deps chromium && \
+    /openhands/micromamba/bin/micromamba run -n openhands poetry run pip install playwright && \
+    /openhands/micromamba/bin/micromamba run -n openhands poetry run playwright install --with-deps chromium && \
     # Set environment variables
-    echo "OH_INTERPRETER_PATH=$(/openhands/miniforge3/bin/mamba run -n base poetry run python -c "import sys; print(sys.executable)")" >> /etc/environment && \
+    echo "OH_INTERPRETER_PATH=$(/openhands/micromamba/bin/micromamba run -n openhands poetry run python -c "import sys; print(sys.executable)")" >> /etc/environment && \
     # Install extra dependencies if specified
     {{ extra_deps }} {% if extra_deps %} && {% endif %} \
     # Clear caches
-    /openhands/miniforge3/bin/mamba run -n base poetry cache clear --all . && \
+    /openhands/micromamba/bin/micromamba run -n openhands poetry cache clear --all . && \
     # Set permissions
     {% if not skip_init %}chmod -R g+rws /openhands/poetry && {% endif %} \
     mkdir -p /openhands/workspace && chmod -R g+rws,o+rw /openhands/workspace && \
     # Clean up
     apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* && \
-    /openhands/miniforge3/bin/mamba clean --all
+    /openhands/micromamba/bin/micromamba clean --all
 # ================================================================
 # END: Copy Project and Install/Update Dependencies
 # ================================================================


Test patch:
diff --git a/.github/workflows/dummy-agent-test.yml b/.github/workflows/dummy-agent-test.yml
index 517af6fe..795391e5 100644
--- a/.github/workflows/dummy-agent-test.yml
+++ b/.github/workflows/dummy-agent-test.yml
@@ -45,7 +45,7 @@ jobs:
       - name: Run tests
         run: |
           set -e
-          poetry run python3 openhands/core/main.py -t "do a flip" -d ./workspace/ -c DummyAgent
+          SANDBOX_FORCE_REBUILD_RUNTIME=True poetry run python3 openhands/core/main.py -t "do a flip" -d ./workspace/ -c DummyAgent
       - name: Check exit code
         run: |
           if [ $? -ne 0 ]; then
diff --git a/tests/runtime/conftest.py b/tests/runtime/conftest.py
index 2308244f..9b5bebab 100644
--- a/tests/runtime/conftest.py
+++ b/tests/runtime/conftest.py
@@ -208,6 +208,7 @@ def _load_runtime(
     base_container_image: str | None = None,
     browsergym_eval_env: str | None = None,
     use_workspace: bool | None = None,
+    force_rebuild_runtime: bool = False,
 ) -> Runtime:
     sid = 'rt_' + str(random.randint(100000, 999999))
 
@@ -217,7 +218,7 @@ def _load_runtime(
 
     config = load_app_config()
     config.run_as_openhands = run_as_openhands
-
+    config.sandbox.force_rebuild_runtime = force_rebuild_runtime
     # Folder where all tests create their own folder
     global test_mount_path
     if use_workspace:
diff --git a/tests/runtime/test_browsing.py b/tests/runtime/test_browsing.py
index c0ea3e18..1d3a4213 100644
--- a/tests/runtime/test_browsing.py
+++ b/tests/runtime/test_browsing.py
@@ -19,7 +19,7 @@ from openhands.events.observation import (
 # Browsing tests
 # ============================================================================================================================
 
-PY3_FOR_TESTING = '/openhands/miniforge3/bin/mamba run -n base python3'
+PY3_FOR_TESTING = '/openhands/micromamba/bin/micromamba run -n openhands python3'
 
 
 def test_simple_browse(temp_dir, box_class, run_as_openhands):
@@ -75,6 +75,7 @@ def test_browsergym_eval_env(box_class, temp_dir):
         run_as_openhands=False,  # need root permission to access file
         base_container_image='xingyaoww/od-eval-miniwob:v1.0',
         browsergym_eval_env='browsergym/miniwob.choose-list',
+        force_rebuild_runtime=True,
     )
     from openhands.runtime.browser.browser_env import (
         BROWSER_EVAL_GET_GOAL_ACTION,
diff --git a/tests/unit/test_runtime_build.py b/tests/unit/test_runtime_build.py
index 0b448f2b..0031f081 100644
--- a/tests/unit/test_runtime_build.py
+++ b/tests/unit/test_runtime_build.py
@@ -155,16 +155,14 @@ def test_generate_dockerfile_scratch():
     )
     assert base_image in dockerfile_content
     assert 'apt-get update' in dockerfile_content
-    assert 'apt-get install -y wget sudo apt-utils' in dockerfile_content
-    assert (
-        'RUN /openhands/miniforge3/bin/mamba install conda-forge::poetry python=3.11 -y'
-        in dockerfile_content
-    )
+    assert 'apt-get install -y wget curl sudo apt-utils' in dockerfile_content
+    assert 'poetry' in dockerfile_content and '-c conda-forge' in dockerfile_content
+    assert 'python=3.11' in dockerfile_content
 
     # Check the update command
     assert 'COPY ./code /openhands/code' in dockerfile_content
     assert (
-        '/openhands/miniforge3/bin/mamba run -n base poetry install'
+        '/openhands/micromamba/bin/micromamba run -n openhands poetry install'
         in dockerfile_content
     )
 
@@ -178,17 +176,13 @@ def test_generate_dockerfile_skip_init():
 
     # These commands SHOULD NOT include in the dockerfile if skip_init is True
     assert 'RUN apt update && apt install -y wget sudo' not in dockerfile_content
-    assert (
-        'RUN /openhands/miniforge3/bin/mamba install conda-forge::poetry python=3.11 -y'
-        not in dockerfile_content
-    )
+    assert '-c conda-forge' not in dockerfile_content
+    assert 'python=3.11' not in dockerfile_content
+    assert 'https://micro.mamba.pm/install.sh' not in dockerfile_content
 
     # These update commands SHOULD still in the dockerfile
     assert 'COPY ./code /openhands/code' in dockerfile_content
-    assert (
-        '/openhands/miniforge3/bin/mamba run -n base poetry install'
-        in dockerfile_content
-    )
+    assert 'poetry install' in dockerfile_content
 
 
 def test_get_runtime_image_repo_and_tag_eventstream():
@@ -353,7 +347,7 @@ def live_docker_image():
     dockerfile_content = f"""
     # syntax=docker/dockerfile:1.4
     FROM {DEFAULT_BASE_IMAGE} AS base
-    RUN apt-get update && apt-get install -y wget sudo apt-utils
+    RUN apt-get update && apt-get install -y wget curl sudo apt-utils
 
     FROM base AS intermediate
     RUN mkdir -p /openhands