Repository: rclone/rclone (language: go)

Issue:
The issue encountered when uploading large size file to Pikpak.
What is the problem you are having with rclone?
I encountered a few days ago while uploading a large file. The file is 134GB, but I encountered the following error before even reaching 100GB:

```bash
2024/05/08 20:56:55 ERROR : /Marvels.Avengers.zip: Failed to copy: failed to upload: MultipartUpload: upload multipart failed
        upload id: C278C4F1FAD2486782772961F3CA663A
caused by: TotalPartsExceeded: exceeded total allowed configured MaxUploadParts (10000). Adjust PartSize to fit in this limit
```

It mentions that adjusting the PartSize parameter can allow the upload. Is the corresponding parameter max-upload-parts? However, I couldn't find this parameter in Pikpak. Currently, I only see two specific parameters for cloud storage services: --oos-max-upload-parts and --s3-max-upload-parts, but neither seems applicable to Pikpak.

I'm not sure what the maximum file size limit for Pikpak is, but currently, uploading large files to Pikpak using rclone seems to be problematic.

If possible, I think rclone could handle large file uploads better. For instance, it could first check if the file size fits within the default PartSize before starting the upload. If it doesn't fit, the upload shouldn't start because uploading large files takes a lot of time. Encountering a PartSize error halfway through, requiring a settings adjustment and re-upload, wastes all the previous time.

Additionally, maybe rclone could automatically adjust the PartSize based on the file size, completing the upload without any errors. I think it's challenging for the average user to know how large the file needs to be before adjusting the PartSize for a successful upload, Of course, these are just my humble suggestions and might be a bit demanding. I'm just offering some possible directions for improvement.

Thank you for your help.

What is your rclone version (output from rclone version)
1.66

Which OS you are using and how many bits (e.g. Windows 7, 64 bit)
Win10 & Ubuntu 22.04, 64 bit

Which cloud storage system are you using? (e.g. Google Drive)
local or google drive upload to pikpak

The command you were trying to run (e.g. rclone copy /tmp remote:tmp)
rclone copy & sync

How to use GitHub
Please use the 👍 reaction to show that you are affected by the same issue.
Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
Subscribe to receive notifications on status change and new comments.


Discussion / hints:
> ```
> caused by: TotalPartsExceeded: exceeded total allowed configured MaxUploadParts (10000). Adjust PartSize to fit in this limit
> ```

We don't currently have a way of adjusting this.

However we do have an internal library which can calculate part sizes for you

https://github.com/rclone/rclone/blob/96e39ea486c4ac2d1955c853b1eeec3e7c2d900a/fs/chunksize/chunksize.go#L20

This needs to go in about here

https://github.com/rclone/rclone/blob/96e39ea486c4ac2d1955c853b1eeec3e7c2d900a/backend/pikpak/pikpak.go#L1170

@wiserain have you got time to take a look at this?
> @wiserain have you got time to take a look at this?

Sure I will make a PR for this after #7852 merged.


Fix patch:
diff --git a/backend/pikpak/pikpak.go b/backend/pikpak/pikpak.go
index 9b6a527cfc368..4a3fbd3d99cec 100644
--- a/backend/pikpak/pikpak.go
+++ b/backend/pikpak/pikpak.go
@@ -18,7 +18,6 @@ package pikpak
 // ------------------------------------------------------------
 
 // * List() with options starred-only
-// * uploadByResumable() with configurable chunk-size
 // * user-configurable list chunk
 // * backend command: untrash, iscached
 // * api(event,task)
@@ -47,6 +46,7 @@ import (
 	"github.com/rclone/rclone/backend/pikpak/api"
 	"github.com/rclone/rclone/fs"
 	"github.com/rclone/rclone/fs/accounting"
+	"github.com/rclone/rclone/fs/chunksize"
 	"github.com/rclone/rclone/fs/config"
 	"github.com/rclone/rclone/fs/config/configmap"
 	"github.com/rclone/rclone/fs/config/configstruct"
@@ -72,6 +72,8 @@ const (
 	waitTime                    = 500 * time.Millisecond
 	decayConstant               = 2 // bigger for slower decay, exponential
 	rootURL                     = "https://api-drive.mypikpak.com"
+	minChunkSize                = fs.SizeSuffix(s3manager.MinUploadPartSize)
+	defaultUploadConcurrency    = s3manager.DefaultUploadConcurrency
 )
 
 // Globals
@@ -193,6 +195,42 @@ Fill in for rclone to use a non root folder as its starting point.
 			Help:     "Files bigger than this will be cached on disk to calculate hash if required.",
 			Default:  fs.SizeSuffix(10 * 1024 * 1024),
 			Advanced: true,
+		}, {
+			Name: "chunk_size",
+			Help: `Chunk size for multipart uploads.
+	
+Large files will be uploaded in chunks of this size.
+
+Note that this is stored in memory and there may be up to
+"--transfers" * "--pikpak-upload-concurrency" chunks stored at once
+in memory.
+
+If you are transferring large files over high-speed links and you have
+enough memory, then increasing this will speed up the transfers.
+
+Rclone will automatically increase the chunk size when uploading a
+large file of known size to stay below the 10,000 chunks limit.
+
+Increasing the chunk size decreases the accuracy of the progress
+statistics displayed with "-P" flag.`,
+			Default:  minChunkSize,
+			Advanced: true,
+		}, {
+			Name: "upload_concurrency",
+			Help: `Concurrency for multipart uploads.
+
+This is the number of chunks of the same file that are uploaded
+concurrently for multipart uploads.
+
+Note that chunks are stored in memory and there may be up to
+"--transfers" * "--pikpak-upload-concurrency" chunks stored at once
+in memory.
+
+If you are uploading small numbers of large files over high-speed links
+and these uploads do not fully utilize your bandwidth, then increasing
+this may help to speed up the transfers.`,
+			Default:  defaultUploadConcurrency,
+			Advanced: true,
 		}, {
 			Name:     config.ConfigEncoding,
 			Help:     config.ConfigEncodingHelp,
@@ -223,6 +261,8 @@ type Options struct {
 	UseTrash            bool                 `config:"use_trash"`
 	TrashedOnly         bool                 `config:"trashed_only"`
 	HashMemoryThreshold fs.SizeSuffix        `config:"hash_memory_limit"`
+	ChunkSize           fs.SizeSuffix        `config:"chunk_size"`
+	UploadConcurrency   int                  `config:"upload_concurrency"`
 	Enc                 encoder.MultiEncoder `config:"encoding"`
 }
 
@@ -432,6 +472,9 @@ func newFs(ctx context.Context, name, path string, m configmap.Mapper) (*Fs, err
 	if err := configstruct.Set(m, opt); err != nil {
 		return nil, err
 	}
+	if opt.ChunkSize < minChunkSize {
+		return nil, fmt.Errorf("chunk size must be at least %s", minChunkSize)
+	}
 
 	root := parsePath(path)
 
@@ -1148,7 +1191,7 @@ func (f *Fs) uploadByForm(ctx context.Context, in io.Reader, name string, size i
 	return
 }
 
-func (f *Fs) uploadByResumable(ctx context.Context, in io.Reader, resumable *api.Resumable) (err error) {
+func (f *Fs) uploadByResumable(ctx context.Context, in io.Reader, name string, size int64, resumable *api.Resumable) (err error) {
 	p := resumable.Params
 	endpoint := strings.Join(strings.Split(p.Endpoint, ".")[1:], ".") // "mypikpak.com"
 
@@ -1161,19 +1204,21 @@ func (f *Fs) uploadByResumable(ctx context.Context, in io.Reader, resumable *api
 	if err != nil {
 		return
 	}
+	partSize := chunksize.Calculator(name, size, s3manager.MaxUploadParts, f.opt.ChunkSize)
 
-	uploader := s3manager.NewUploader(sess)
+	// Create an uploader with the session and custom options
+	uploader := s3manager.NewUploader(sess, func(u *s3manager.Uploader) {
+		u.PartSize = int64(partSize)
+		u.Concurrency = f.opt.UploadConcurrency
+	})
 	// Upload input parameters
 	uParams := &s3manager.UploadInput{
 		Bucket: &p.Bucket,
 		Key:    &p.Key,
 		Body:   in,
 	}
-	// Perform upload with options different than the those in the Uploader.
-	_, err = uploader.UploadWithContext(ctx, uParams, func(u *s3manager.Uploader) {
-		// TODO can be user-configurable
-		u.PartSize = 10 * 1024 * 1024 // 10MB part size
-	})
+	// Perform an upload
+	_, err = uploader.UploadWithContext(ctx, uParams)
 	return
 }
 
@@ -1224,7 +1269,7 @@ func (f *Fs) upload(ctx context.Context, in io.Reader, leaf, dirID, sha1Str stri
 	if uploadType == api.UploadTypeForm && new.Form != nil {
 		err = f.uploadByForm(ctx, in, req.Name, size, new.Form, options...)
 	} else if uploadType == api.UploadTypeResumable && new.Resumable != nil {
-		err = f.uploadByResumable(ctx, in, new.Resumable)
+		err = f.uploadByResumable(ctx, in, leaf, size, new.Resumable)
 	} else {
 		err = fmt.Errorf("no method available for uploading: %+v", new)
 	}


Test patch: