Repository: strapi/strapi

Issue #17557: Transfer between dockerized instances fails
Created: 2023-08-03T08:57:46Z
Bug report
Required System information


Production/Staging:

Node.js version: v16.20.1
NPM version: 8.19.4
Strapi version: 4.12.1
Database: mysql  Ver 15.1 Distrib 10.4.28-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2
Operating system: Ubuntu 22.04.2 (with Docker version 23.0.3, build 3e7cbfd)
Is your project Javascript or Typescript: JS

Local Env:

Node.js version: v16.14.2
NPM version: 8.12.2
Strapi version: 4.12.1
Database: sqlite
Operating system: MacOS 13.4.1
Is your project Javascript or Typescript: JS

Describe the bug
Strapi transfer was fixed in v4.12.1. 🥳
It worked perfectly to pull data from production (dockerized setup with mysql database) to a local dev environment (barebone node with sqlite). And it worked to push the same data from dev to staging (dockerized setup with mysql database).
But directly transfering between production and staging fails:
? The transfer will delete all the local Strapi assets and its database. Are you sure you want to proceed? Yes
Starting transfer...
✔ entities: 811 transfered (size: 1.2 MB) (elapsed: 4946 ms)
⠧ assets: 1053 transfered (size: 73.7 MB) (elapsed: 30066 ms) (2.5 MB/s)/opt/node_modules/@strapi/data-transfer/dist/strapi/providers/remote-source/index.js:59
                    await this.writeAsync(assets[item.assetID].stream, chunk);
                                                               ^

TypeError: Cannot read properties of undefined (reading 'stream')
    at PassThrough.<anonymous> (/opt/node_modules/@strapi/data-transfer/dist/strapi/providers/remote-source/index.js:59:64)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
I first tried to work around that by excluding the files first and then to try to move the files separately. Here it fails as well, but the error message is a little bit more descriptive:
me@staging:~$ docker compose exec -it strapi strapi transfer --from-token $TOKEN --from https://STRAPI/admin/ --force --only files
? The transfer will delete all the local Strapi assets and its database. Are you sure you want to proceed? Yes
Starting transfer...
✖ assets: 26 transfered (size: 303.9 KB) (elapsed: 28 ms)
[2023-08-02 23:16:14.677] error: [ERROR] Error while uploading asset logo_white_27348633e8.svg TypeError: Cannot set properties of null (setting 'url')
Transfer process failed.

Any ideas?
As mentioned before, I was able to transfer the data from production to staging via my local dev environment.
Steps to reproduce the behavior
Not sure if this can be reproduced on a clean instance. I'll gladly add some debug statements if that helps.
Expected behavior
Transfer should work.

Duplicate issue #17809: 'yarn strapi import' doesn't work in Docker (permissions on the public directory)
Created: 2023-08-25T15:58:36Z
Bug report
FYI : I'm using Docker
Required System information



Node.js version:

v18.17.1


NPM version:

v9.6.7
but I use yarn v1.22.19


Strapi version:

v4.12.6


Database:

psql (PostgreSQL) 12.0


Operating system:

macOS Ventura 13.5.1 with Docker


Is your project Javascript or Typescript:

TypeScript



Describe the bug
The bug is about the Strapi import with the command yarn strapi import. A fatal error occurs when I launch this command inside Docker.
error: [FATAL] The backup folder for the assets could not be created inside the public folder. Please ensure Strapi has write permissions on the public directory Import process failed.
Steps to reproduce the behavior
In CLI :

Delete all container, images, and volumes
Install via Docker-compose the project :
docker-compose up strapi --force-recreate
Add an image in Media library
Create just one entry with this image
Export datas :
docker exec strapi yarn strapi export --file test --key test
Delete the image and the entry
Import the datas previously exported :
docker exec -it strapi /bin/bash
yarn strapi import -f test.tar.gz.enc --key test
See error

yarn run v1.22.19
$ strapi import -f test.tar.gz.enc --key test
? The import will delete all assets and data in your database. Are you sure you want to proceed?
Yes
Starting the compilation for TypeScript files in /opt/app
Starting import...
[2023-08-25 14:10:48.545] error: [FATAL] The backup folder for the assets could not be created inside the public folder. Please ensure Strapi has write permissions on the public directory
Import process failed.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
Expected behavior
After this last command, datas (like entries, media and settings) are imported correctly.
Screenshots
...
Code snippets
These are my docker and docker-compose files and the export :
docker :
FROM node:18-alpine
# Installing libvips-dev for sharp Compatibility
RUN apk update && apk add --no-cache build-base gcc autoconf automake zlib-dev libpng-dev nasm bash vips-dev
ARG NODE_ENV=development
ENV NODE_ENV=${NODE_ENV}

WORKDIR /opt/
COPY package.json yarn.lock ./
RUN yarn config set network-timeout 600000 -g && yarn install
ENV PATH /opt/node_modules/.bin:$PATH

WORKDIR /opt/app
COPY . .
RUN chown -R node:node /opt/app
USER node
RUN ["yarn", "build"]
EXPOSE 1337
CMD ["yarn", "develop"]

docker-compose :
version: "3"
services:
  strapi:
    container_name: strapi
    build:
      context: .
      dockerfile: Dockerfile
    image: strapi:latest
    restart: unless-stopped
    env_file: .env
    environment:
      DATABASE_CLIENT: ${DATABASE_CLIENT}
      DATABASE_HOST: strapiDB
      DATABASE_PORT: ${DATABASE_PORT}
      DATABASE_NAME: ${DATABASE_NAME}
      DATABASE_USERNAME: ${DATABASE_USERNAME}
      DATABASE_PASSWORD: ${DATABASE_PASSWORD}
      JWT_SECRET: ${JWT_SECRET}
      ADMIN_JWT_SECRET: ${ADMIN_JWT_SECRET}
      APP_KEYS: ${APP_KEYS}
      NODE_ENV: ${NODE_ENV}
    volumes:
      - ./config:/opt/app/config
      - ./src:/opt/app/src
      - ./package.json:/opt/package.json
      - ./yarn.lock:/opt/yarn.lock
      - ./.env:/opt/app/.env
      - ./public/uploads:/opt/app/public/uploads
    ports:
      - "1337:1337"
    networks:
      - strapi
    depends_on:
      - strapiDB

  strapiDB:
    container_name: strapiDB
    platform: linux/amd64 #for platform error on Apple M1 chips
    restart: unless-stopped
    env_file: .env
    image: postgres:12.0-alpine
    environment:
      POSTGRES_USER: ${DATABASE_USERNAME}
      POSTGRES_PASSWORD: ${DATABASE_PASSWORD}
      POSTGRES_DB: ${DATABASE_NAME}
    volumes:
      - strapi-data:/var/lib/postgresql/data/ #using a volume
      #- ./data:/var/lib/postgresql/data/ # if you want to use a bind folder

    ports:
      - "5432:5432"
    networks:
      - strapi

volumes:
  strapi-data:

networks:
  strapi:
    name: Strapi
    driver: bridge

export successful :
docker exec strapi yarn strapi export --file test --key test
yarn run v1.22.19
$ strapi export --file test --key test
Starting the compilation for TypeScript files in /opt/app
Starting export...
- schemas: 0 transfered (size: 0) (elapsed: 0 ms)
✔ schemas: 26 transfered (size: 40.2 KB) (elapsed: 5 ms)
- entities: 0 transfered (size: 0) (elapsed: 0 ms)
✔ entities: 15 transfered (size: 4.4 KB) (elapsed: 57 ms)
- assets: 0 transfered (size: 0) (elapsed: 0 ms)
✔ assets: 5 transfered (size: 808.8 KB) (elapsed: 66 ms)
- links: 0 transfered (size: 0) (elapsed: 0 ms)
✔ links: 11 transfered (size: 2.2 KB) (elapsed: 88 ms)
- configuration: 0 transfered (size: 0) (elapsed: 0 ms)
✔ configuration: 38 transfered (size: 126.3 KB) (elapsed: 5 ms)
┌─────────────────────────────────────────┬───────┬───────────────┐
│ Type                                    │ Count │ Size          │
├─────────────────────────────────────────┼───────┼───────────────┤
│ schemas                                 │    26 │      40.2 KB  │
├─────────────────────────────────────────┼───────┼───────────────┤
│ -- contentType                          │    26 │ (    40.2 KB) │
├─────────────────────────────────────────┼───────┼───────────────┤
│ entities                                │    15 │       4.4 KB  │
├─────────────────────────────────────────┼───────┼───────────────┤
│ -- api::gallery.gallery                 │     1 │ (     239 B ) │
├─────────────────────────────────────────┼───────┼───────────────┤
│ -- plugin::i18n.locale                  │     2 │ (     315 B ) │
├─────────────────────────────────────────┼───────┼───────────────┤
│ -- plugin::upload.file                  │     1 │ (     1.6 KB) │
├─────────────────────────────────────────┼───────┼───────────────┤
│ -- plugin::users-permissions.permission │     9 │ (     1.7 KB) │
├─────────────────────────────────────────┼───────┼───────────────┤
│ -- plugin::users-permissions.role       │     2 │ (     466 B ) │
├─────────────────────────────────────────┼───────┼───────────────┤
│ assets                                  │     5 │     808.8 KB  │
├─────────────────────────────────────────┼───────┼───────────────┤
│ -- .png                                 │     5 │ (   808.8 KB) │
├─────────────────────────────────────────┼───────┼───────────────┤
│ links                                   │    11 │       2.2 KB  │
├─────────────────────────────────────────┼───────┼───────────────┤
│ configuration                           │    38 │     126.3 KB  │
├─────────────────────────────────────────┼───────┼───────────────┤
│ Total                                   │    95 │     981.9 KB  │
└─────────────────────────────────────────┴───────┴───────────────┘
Export archive is in test.tar.gz.enc
Export process has been completed successfully!
Done in 4.70s.
Additional context
I tried to change chmod on the folder "public" or "uploads" with different status but it doesn't work

Closing comment (2023-12-11T12:05:46Z): @derrickmehaffy : This does not seem to be a duplicate of  #17557
The problem described here originates from the import code trying to move the whole /opt/app/public/uploads folder to a backup destination.
Since a typical setup has the upload folder as a mounted volume inside the docker container, it's not possible to call rename() on that folder.
See the relevant part of the output of  strace -f yarn strapi import --force -f /path/to/data.tar.gz if /opt/app/public/uploads is a docker volume:
[pid    41] rename("/opt/app/public/uploads", "/opt/app/public/uploads_backup_1702295200814" <unfinished ...>
[pid    41] <... rename resumed>)       = -1 EBUSY (Resource busy)

We double-checked that no other process is using /opt/app/public/uploads at that time so EBUSY is returned because it is a mounted volume.
P.S.: A workaround when  experiencing this problem (@Ineslujan , @zeitiv ) is: mount the volume at /opt/app/public/ if possible