Repository: dragonflyoss/Dragonfly2
Pull request #854 (2021-12-02 09:40:25)

unify piece size
<!--- Provide a general summary of your changes in the Title above -->

## Description

<!--- Describe your changes in detail -->

## Related Issue

<!--- This project only accepts pull requests related to open issues -->
<!--- If suggesting a new feature or change, please discuss it in an issue first -->
<!--- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!--- Please link to the issue here: -->

## Motivation and Context

<!--- Why is this change required? What problem does it solve? -->

## Screenshots (if appropriate):

## Types of changes

<!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->

- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
- [ ] Documentation Update (if none of the other choices apply)

## Checklist:

<!--- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->

- [ ] Code compiles correctly.
- [ ] My code follows the code style of this project.
- [ ] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [ ] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [ ] All new and existing tests passed.


codecov-commenter: # [Codecov](https://codecov.io/gh/dragonflyoss/Dragonfly2/pull/854?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss) Report
> Merging [#854](https://codecov.io/gh/dragonflyoss/Dragonfly2/pull/854?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss) (4746cdf) into [main](https://codecov.io/gh/dragonflyoss/Dragonfly2/commit/b122200937fa3da78f324fc76cc0e958047a80ad?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss) (b122200) will **increase** coverage by `0.03%`.
> The diff coverage is `50.00%`.

[![Impacted file tree graph](https://codecov.io/gh/dragonflyoss/Dragonfly2/pull/854/graphs/tree.svg?width=650&height=150&src=pr&token=PUICJYSAT7&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss)](https://codecov.io/gh/dragonflyoss/Dragonfly2/pull/854?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss)

```diff
@@            Coverage Diff             @@
##             main     #854      +/-   ##
==========================================
+ Coverage   41.29%   41.33%   +0.03%     
==========================================
  Files         102      103       +1     
  Lines        8040     8047       +7     
==========================================
+ Hits         3320     3326       +6     
- Misses       4309     4310       +1     
  Partials      411      411              
```

| Flag | Coverage Δ | |
|---|---|---|
| Object-compatibility-e2etests | `?` | |
| e2etests | `?` | |
| unittests | `41.33% <50.00%> (+0.03%)` | :arrow_up: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss#carryforward-flags-in-the-pull-request-comment) to find out more.

| [Impacted Files](https://codecov.io/gh/dragonflyoss/Dragonfly2/pull/854?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss) | Coverage Δ | |
|---|---|---|
| [cdn/supervisor/task/manager\_util.go](https://codecov.io/gh/dragonflyoss/Dragonfly2/pull/854/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss#diff-Y2RuL3N1cGVydmlzb3IvdGFzay9tYW5hZ2VyX3V0aWwuZ28=) | `0.00% <0.00%> (ø)` | |
| [internal/dfutils/dfutils.go](https://codecov.io/gh/dragonflyoss/Dragonfly2/pull/854/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss#diff-aW50ZXJuYWwvZGZ1dGlscy9kZnV0aWxzLmdv) | `100.00% <ø> (ø)` | |
| [client/daemon/peer/piece\_manager.go](https://codecov.io/gh/dragonflyoss/Dragonfly2/pull/854/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss#diff-Y2xpZW50L2RhZW1vbi9wZWVyL3BpZWNlX21hbmFnZXIuZ28=) | `67.52% <100.00%> (ø)` | |
| [client/daemon/peer/peertask\_base.go](https://codecov.io/gh/dragonflyoss/Dragonfly2/pull/854/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss#diff-Y2xpZW50L2RhZW1vbi9wZWVyL3BlZXJ0YXNrX2Jhc2UuZ28=) | `45.45% <0.00%> (-1.22%)` | :arrow_down: |
| [client/daemon/peer/peertask\_stream.go](https://codecov.io/gh/dragonflyoss/Dragonfly2/pull/854/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss#diff-Y2xpZW50L2RhZW1vbi9wZWVyL3BlZXJ0YXNrX3N0cmVhbS5nbw==) | `55.35% <0.00%> (+1.52%)` | :arrow_up: |


Patch:
diff --git a/api/manager/docs.go b/api/manager/docs.go
index 611ab211..e72ce9a0 100644
--- a/api/manager/docs.go
+++ b/api/manager/docs.go
@@ -4298,7 +4298,18 @@ var doc = `{
             "type": "object"
         },
         "types.SchedulerClusterScopes": {
-            "type": "object"
+            "type": "object",
+            "properties": {
+                "idc": {
+                    "type": "string"
+                },
+                "location": {
+                    "type": "string"
+                },
+                "net_topology": {
+                    "type": "string"
+                }
+            }
         },
         "types.SignUpRequest": {
             "type": "object",
diff --git a/api/manager/swagger.json b/api/manager/swagger.json
index 8eaf15a9..955e4808 100644
--- a/api/manager/swagger.json
+++ b/api/manager/swagger.json
@@ -4284,7 +4284,18 @@
             "type": "object"
         },
         "types.SchedulerClusterScopes": {
-            "type": "object"
+            "type": "object",
+            "properties": {
+                "idc": {
+                    "type": "string"
+                },
+                "location": {
+                    "type": "string"
+                },
+                "net_topology": {
+                    "type": "string"
+                }
+            }
         },
         "types.SignUpRequest": {
             "type": "object",
diff --git a/api/manager/swagger.yaml b/api/manager/swagger.yaml
index d18a4d92..3d20e07f 100644
--- a/api/manager/swagger.yaml
+++ b/api/manager/swagger.yaml
@@ -558,6 +558,13 @@ definitions:
   types.SchedulerClusterConfig:
     type: object
   types.SchedulerClusterScopes:
+    properties:
+      idc:
+        type: string
+      location:
+        type: string
+      net_topology:
+        type: string
     type: object
   types.SignUpRequest:
     properties:
diff --git a/cdn/cdn.go b/cdn/cdn.go
index a2e78a04..8e459cd1 100644
--- a/cdn/cdn.go
+++ b/cdn/cdn.go
@@ -158,7 +158,7 @@ func (s *Server) Serve() error {
 				if err == http.ErrServerClosed {
 					return
 				}
-				logger.Fatalf("metrics server closed unexpect: %+v", err)
+				logger.Fatalf("metrics server closed unexpect: %#v", err)
 			}
 		}()
 	}
@@ -182,14 +182,14 @@ func (s *Server) Serve() error {
 	}
 	lis, _, err := rpc.ListenWithPortRange(listen, s.config.ListenPort, s.config.ListenPort)
 	if err != nil {
-		logger.Fatalf("net listener failed to start: %+v", err)
+		logger.Fatalf("net listener failed to start: %v", err)
 	}
 	defer lis.Close()
 
 	// Started GRPC server
 	logger.Infof("started grpc server at %s://%s", lis.Addr().Network(), lis.Addr().String())
 	if err := s.grpcServer.Serve(lis); err != nil {
-		logger.Errorf("stoped grpc server: %+v", err)
+		logger.Errorf("stoped grpc server: %v", err)
 		return err
 	}
 
@@ -206,7 +206,7 @@ func (s *Server) Stop() {
 	// Stop metrics server
 	if s.metricsServer != nil {
 		if err := s.metricsServer.Shutdown(context.Background()); err != nil {
-			logger.Errorf("metrics server failed to stop: %+v", err)
+			logger.Errorf("metrics server failed to stop: %#v", err)
 		}
 		logger.Info("metrics server closed under request")
 	}
diff --git a/cdn/config/constants.go b/cdn/config/constants.go
index 0106c77c..3829fb2b 100644
--- a/cdn/config/constants.go
+++ b/cdn/config/constants.go
@@ -37,14 +37,6 @@ const (
 	DefaultStorageMode = "disk"
 )
 
-const (
-	// DefaultPieceSize 4M
-	DefaultPieceSize = 4 * 1024 * 1024
-
-	// DefaultPieceSizeLimit 15M
-	DefaultPieceSizeLimit = 15 * 1024 * 1024
-)
-
 const (
 	// DefaultFailAccessInterval is the interval time after failed to access the URL.
 	DefaultFailAccessInterval = 3 * time.Minute
diff --git a/cdn/rpcserver/rpcserver.go b/cdn/rpcserver/rpcserver.go
index cbf37f67..c706ef0c 100644
--- a/cdn/rpcserver/rpcserver.go
+++ b/cdn/rpcserver/rpcserver.go
@@ -87,7 +87,7 @@ func (css *server) ObtainSeeds(ctx context.Context, req *cdnsystem.SeedRequest,
 	defer span.End()
 	span.SetAttributes(config.AttributeObtainSeedsRequest.String(req.String()))
 	span.SetAttributes(config.AttributeTaskID.String(req.TaskId))
-	logger.Infof("obtain seeds request: %+v", req)
+	logger.Infof("obtain seeds request: %#v", req)
 	defer func() {
 		if r := recover(); r != nil {
 			err = dferrors.Newf(base.Code_UnknownError, "obtain task(%s) seeds encounter an panic: %v", req.TaskId, r)
@@ -159,7 +159,7 @@ func (css *server) GetPieceTasks(ctx context.Context, req *base.PieceTaskRequest
 			logger.WithTaskID(req.TaskId).Errorf("%v", err)
 		}
 	}()
-	logger.Infof("get piece tasks: %+v", req)
+	logger.Infof("get piece tasks: %#v", req)
 	task, err := css.taskMgr.Get(req.TaskId)
 	if err != nil {
 		if cdnerrors.IsDataNotFound(err) {
diff --git a/cdn/supervisor/cdn/cache_detector.go b/cdn/supervisor/cdn/cache_detector.go
index 0bf0c086..6094394b 100644
--- a/cdn/supervisor/cdn/cache_detector.go
+++ b/cdn/supervisor/cdn/cache_detector.go
@@ -52,7 +52,7 @@ type cacheResult struct {
 }
 
 func (s *cacheResult) String() string {
-	return fmt.Sprintf("{breakNum: %d, pieceMetaRecords: %+v, fileMetaData: %+v}", s.breakPoint, s.pieceMetaRecords, s.fileMetaData)
+	return fmt.Sprintf("{breakNum: %d, pieceMetaRecords: %#v, fileMetaData: %#v}", s.breakPoint, s.pieceMetaRecords, s.fileMetaData)
 }
 
 // newCacheDetector create a new cache detector
diff --git a/cdn/supervisor/cdn/downloader.go b/cdn/supervisor/cdn/downloader.go
index 58e08d62..c6ce3609 100644
--- a/cdn/supervisor/cdn/downloader.go
+++ b/cdn/supervisor/cdn/downloader.go
@@ -43,7 +43,7 @@ func (cm *Manager) download(ctx context.Context, task *types.SeedTask, detectRes
 			headers[RangeHeaderName] = fmt.Sprintf("bytes=%s", breakRange)
 		}
 	}
-	task.Log().Infof("start download url %s at range: %d-%d: with header: %+v", task.URL, detectResult.breakPoint,
+	task.Log().Infof("start download url %s at range: %d-%d: with header: %#v", task.URL, detectResult.breakPoint,
 		task.SourceFileLength, task.Header)
 	reader, responseHeader, err := source.DownloadWithResponseHeader(ctx, task.URL, headers)
 	// update Expire info
diff --git a/cdn/supervisor/cdn/manager.go b/cdn/supervisor/cdn/manager.go
index 13ad0aa4..cf24afeb 100644
--- a/cdn/supervisor/cdn/manager.go
+++ b/cdn/supervisor/cdn/manager.go
@@ -110,7 +110,7 @@ func (cm *Manager) TriggerCDN(ctx context.Context, task *types.SeedTask) (seedTa
 		return seedTask, errors.Wrapf(err, "failed to detect cache")
 	}
 	span.SetAttributes(config.AttributeCacheResult.String(detectResult.String()))
-	task.Log().Debugf("detects cache result: %+v", detectResult)
+	task.Log().Debugf("detects cache result: %#v", detectResult)
 	// second: report detect result
 	err = cm.cdnReporter.reportCache(ctx, task.TaskID, detectResult)
 	if err != nil {
@@ -176,7 +176,7 @@ func (cm *Manager) TryFreeSpace(fileLength int64) (bool, error) {
 }
 
 func (cm *Manager) handleCDNResult(task *types.SeedTask, sourceDigest string, downloadMetadata *downloadMetadata) (bool, error) {
-	task.Log().Debugf("handle cdn result, downloadMetaData: %+v", downloadMetadata)
+	task.Log().Debugf("handle cdn result, downloadMetaData: %#v", downloadMetadata)
 	var isSuccess = true
 	var errorMsg string
 	// check md5
@@ -219,7 +219,7 @@ func (cm *Manager) handleCDNResult(task *types.SeedTask, sourceDigest string, do
 		return false, errors.New(errorMsg)
 	}
 
-	task.Log().Infof("success to get task, downloadMetadata: %+v realDigest: %s", downloadMetadata, sourceDigest)
+	task.Log().Infof("success to get task, downloadMetadata: %#v realDigest: %s", downloadMetadata, sourceDigest)
 
 	return true, nil
 }
diff --git a/cdn/supervisor/cdn/storage/storage_gc.go b/cdn/supervisor/cdn/storage/storage_gc.go
index 86c411dc..6e36eeaa 100644
--- a/cdn/supervisor/cdn/storage/storage_gc.go
+++ b/cdn/supervisor/cdn/storage/storage_gc.go
@@ -119,7 +119,7 @@ func (cleaner *Cleaner) GC(storagePattern string, force bool) ([]string, error)
 		}
 		// put taskId into gapTasks or intervalTasks which will sort by some rules
 		if err := cleaner.sortInert(gapTasks, intervalTasks, metaData); err != nil {
-			logger.GcLogger.With("type", storagePattern).Errorf("failed to parse inert metaData(%+v): %v", metaData, err)
+			logger.GcLogger.With("type", storagePattern).Errorf("failed to parse inert metaData(%#v): %v", metaData, err)
 		}
 
 		return nil
diff --git a/cdn/supervisor/progress/manager.go b/cdn/supervisor/progress/manager.go
index 020da131..d4dac827 100644
--- a/cdn/supervisor/progress/manager.go
+++ b/cdn/supervisor/progress/manager.go
@@ -104,7 +104,7 @@ func (pm *Manager) WatchSeedProgress(ctx context.Context, taskID string) (<-chan
 	ele := chanList.PushBack(ch)
 	go func(seedCh chan *types.SeedPiece, ele *list.Element) {
 		for _, pieceMetaRecord := range pieceMetaDataRecords {
-			logger.Debugf("seed piece meta record %+v", pieceMetaRecord)
+			logger.Debugf("seed piece meta record %#v", pieceMetaRecord)
 			select {
 			case seedCh <- pieceMetaRecord:
 			case <-time.After(pm.timeout):
@@ -123,7 +123,7 @@ func (pm *Manager) PublishPiece(ctx context.Context, taskID string, record *type
 	span := trace.SpanFromContext(ctx)
 	recordBytes, _ := json.Marshal(record)
 	span.AddEvent(config.EventPublishPiece, trace.WithAttributes(config.AttributeSeedPiece.String(string(recordBytes))))
-	logger.Debugf("seed piece meta record %+v", record)
+	logger.Debugf("seed piece meta record %#v", record)
 	pm.mu.Lock(taskID, false)
 	defer pm.mu.UnLock(taskID, false)
 	err := pm.setPieceMetaRecord(taskID, record)
@@ -155,7 +155,7 @@ func (pm *Manager) PublishTask(ctx context.Context, taskID string, task *types.S
 	span := trace.SpanFromContext(ctx)
 	taskBytes, _ := json.Marshal(task)
 	span.AddEvent(config.EventPublishTask, trace.WithAttributes(config.AttributeSeedTask.String(string(taskBytes))))
-	logger.Debugf("publish task record %+v", task)
+	logger.Debugf("publish task record %#v", task)
 	pm.mu.Lock(taskID, false)
 	defer pm.mu.UnLock(taskID, false)
 	chanList, err := pm.seedSubscribers.GetAsList(taskID)
diff --git a/cdn/supervisor/task/manager.go b/cdn/supervisor/task/manager.go
index 7ae54d18..ddcdefc5 100644
--- a/cdn/supervisor/task/manager.go
+++ b/cdn/supervisor/task/manager.go
@@ -79,12 +79,12 @@ func (tm *Manager) Register(ctx context.Context, req *types.TaskRegisterRequest)
 	task, err := tm.addOrUpdateTask(ctx, req)
 	if err != nil {
 		span.RecordError(err)
-		logger.WithTaskID(req.TaskID).Infof("failed to add or update task with req: %+v: %v", req, err)
+		logger.WithTaskID(req.TaskID).Infof("failed to add or update task with req: %#v: %v", req, err)
 		return nil, err
 	}
 	taskBytes, _ := json.Marshal(task)
 	span.SetAttributes(config.AttributeTaskInfo.String(string(taskBytes)))
-	task.Log().Debugf("success get task info: %+v", task)
+	task.Log().Debugf("success get task info: %#v", task)
 
 	// update accessTime for taskId
 	if err := tm.accessTimeMap.Add(task.TaskID, time.Now()); err != nil {
@@ -148,7 +148,7 @@ func (tm *Manager) triggerCdnSyncAction(ctx context.Context, task *types.SeedTas
 		if err != nil {
 			task.Log().Errorf("failed to update task: %v", err)
 		}
-		task.Log().Infof("successfully update task cdn updatedTask: %+v", updatedTask)
+		task.Log().Infof("successfully update task cdn updatedTask: %#v", updatedTask)
 	}()
 	return nil
 }
@@ -169,7 +169,7 @@ func (tm *Manager) getTask(taskID string) (*types.SeedTask, error) {
 	if info, ok := v.(*types.SeedTask); ok {
 		return info, nil
 	}
-	return nil, errors.Wrapf(cdnerrors.ErrConvertFailed, "origin object: %+v", v)
+	return nil, errors.Wrapf(cdnerrors.ErrConvertFailed, "origin object: %#v", v)
 }
 
 func (tm Manager) Get(taskID string) (*types.SeedTask, error) {
diff --git a/cdn/supervisor/task/manager_util.go b/cdn/supervisor/task/manager_util.go
index 40ae4e1a..f8bdf86e 100644
--- a/cdn/supervisor/task/manager_util.go
+++ b/cdn/supervisor/task/manager_util.go
@@ -25,11 +25,11 @@ import (
 	"github.com/pkg/errors"
 	"go.opentelemetry.io/otel/trace"
 
-	"d7y.io/dragonfly/v2/cdn/cdnutil"
 	"d7y.io/dragonfly/v2/cdn/config"
 	cdnerrors "d7y.io/dragonfly/v2/cdn/errors"
 	"d7y.io/dragonfly/v2/cdn/types"
 	logger "d7y.io/dragonfly/v2/internal/dflog"
+	"d7y.io/dragonfly/v2/internal/dfutils"
 	"d7y.io/dragonfly/v2/pkg/source"
 	"d7y.io/dragonfly/v2/pkg/synclock"
 	"d7y.io/dragonfly/v2/pkg/util/net/urlutils"
@@ -70,8 +70,8 @@ func (tm *Manager) addOrUpdateTask(ctx context.Context, request *types.TaskRegis
 		span.SetAttributes(config.AttributeIfReuseTask.Bool(true))
 		existTask := v.(*types.SeedTask)
 		if !isSameTask(existTask, newTask) {
-			span.RecordError(fmt.Errorf("newTask: %+v, existTask: %+v", newTask, existTask))
-			return nil, cdnerrors.ErrTaskIDDuplicate{TaskID: taskID, Cause: fmt.Errorf("newTask: %+v, existTask: %+v", newTask, existTask)}
+			span.RecordError(fmt.Errorf("newTask: %#v, existTask: %#v", newTask, existTask))
+			return nil, cdnerrors.ErrTaskIDDuplicate{TaskID: taskID, Cause: fmt.Errorf("newTask: %#v, existTask: %#v", newTask, existTask)}
 		}
 		task = existTask
 		logger.Debugf("get exist task for taskID: %s", taskID)
@@ -118,14 +118,14 @@ func (tm *Manager) addOrUpdateTask(ctx context.Context, request *types.TaskRegis
 
 	// calculate piece size and update the PieceSize and PieceTotal
 	if task.PieceSize <= 0 {
-		pieceSize := cdnutil.ComputePieceSize(task.SourceFileLength)
+		pieceSize := dfutils.ComputePieceSize(task.SourceFileLength)
 		task.PieceSize = int32(pieceSize)
 	}
 	if err := tm.taskStore.Add(task.TaskID, task); err != nil {
 		return nil, err
 	}
 
-	logger.Debugf("success add task: %+v into taskStore", task)
+	logger.Debugf("success add task: %#v into taskStore", task)
 	return task, nil
 }
 
diff --git a/client/daemon/peer/piece_manager.go b/client/daemon/peer/piece_manager.go
index fa42b0e4..7ccfd1e9 100644
--- a/client/daemon/peer/piece_manager.go
+++ b/client/daemon/peer/piece_manager.go
@@ -24,11 +24,11 @@ import (
 
 	"golang.org/x/time/rate"
 
-	"d7y.io/dragonfly/v2/cdn/cdnutil"
 	"d7y.io/dragonfly/v2/client/clientutil"
 	"d7y.io/dragonfly/v2/client/config"
 	"d7y.io/dragonfly/v2/client/daemon/storage"
 	logger "d7y.io/dragonfly/v2/internal/dflog"
+	"d7y.io/dragonfly/v2/internal/dfutils"
 	"d7y.io/dragonfly/v2/pkg/rpc/base"
 	"d7y.io/dragonfly/v2/pkg/rpc/scheduler"
 	"d7y.io/dragonfly/v2/pkg/source"
@@ -55,7 +55,7 @@ var _ PieceManager = (*pieceManager)(nil)
 func NewPieceManager(s storage.TaskStorageDriver, pieceDownloadTimeout time.Duration, opts ...func(*pieceManager)) (PieceManager, error) {
 	pm := &pieceManager{
 		storageManager:   s,
-		computePieceSize: cdnutil.ComputePieceSize,
+		computePieceSize: dfutils.ComputePieceSize,
 		calculateDigest:  true,
 	}
 	for _, opt := range opts {
diff --git a/docs/en/api-reference/api-reference.md b/docs/en/api-reference/api-reference.md
index 38bf3422..625f59ea 100644
--- a/docs/en/api-reference/api-reference.md
+++ b/docs/en/api-reference/api-reference.md
@@ -3869,7 +3869,12 @@ delete role by uri config
 
 <a name="types-schedulerclusterscopes"></a>
 ### types.SchedulerClusterScopes
-*Type* : object
+
+|Name|Schema|
+|---|---|
+|**idc**  <br>*optional*|string|
+|**location**  <br>*optional*|string|
+|**net_topology**  <br>*optional*|string|
 
 
 <a name="types-signuprequest"></a>
diff --git a/docs/zh-CN/api-reference/api-reference.md b/docs/zh-CN/api-reference/api-reference.md
index d408622a..035e03cf 100644
--- a/docs/zh-CN/api-reference/api-reference.md
+++ b/docs/zh-CN/api-reference/api-reference.md
@@ -3869,7 +3869,12 @@ delete role by uri config
 
 <a name="types-schedulerclusterscopes"></a>
 ### types.SchedulerClusterScopes
-*类型* : object
+
+|名称|类型|
+|---|---|
+|**idc**  <br>*可选*|string|
+|**location**  <br>*可选*|string|
+|**net_topology**  <br>*可选*|string|
 
 
 <a name="types-signuprequest"></a>
diff --git a/cdn/cdnutil/cdn_util.go b/internal/dfutils/dfutils.go
similarity index 70%
rename from cdn/cdnutil/cdn_util.go
rename to internal/dfutils/dfutils.go
index a2a628f0..236c773e 100644
--- a/cdn/cdnutil/cdn_util.go
+++ b/internal/dfutils/dfutils.go
@@ -14,25 +14,29 @@
  * limitations under the License.
  */
 
-package cdnutil
+package dfutils
 
-import (
-	"d7y.io/dragonfly/v2/cdn/config"
+const (
+	// DefaultPieceSize 4M
+	DefaultPieceSize = 4 * 1024 * 1024
+
+	// DefaultPieceSizeLimit 15M
+	DefaultPieceSizeLimit = 15 * 1024 * 1024
 )
 
 // ComputePieceSize computes the piece size with specified fileLength.
 //
-// If the fileLength<=0, which means failed to get fileLength
+// If the fileLength<0, which means failed to get fileLength
 // and then use the DefaultPieceSize.
 func ComputePieceSize(length int64) uint32 {
 	if length <= 0 || length <= 200*1024*1024 {
-		return config.DefaultPieceSize
+		return DefaultPieceSize
 	}
 
 	gapCount := length / int64(100*1024*1024)
-	mpSize := (gapCount-2)*1024*1024 + config.DefaultPieceSize
-	if mpSize > config.DefaultPieceSizeLimit {
-		return config.DefaultPieceSizeLimit
+	mpSize := (gapCount-2)*1024*1024 + DefaultPieceSize
+	if mpSize > DefaultPieceSizeLimit {
+		return DefaultPieceSizeLimit
 	}
 	return uint32(mpSize)
 }
diff --git a/manager/manager.go b/manager/manager.go
index 0d34209d..6150342e 100644
--- a/manager/manager.go
+++ b/manager/manager.go
@@ -127,7 +127,7 @@ func (s *Server) Serve() error {
 			if err == http.ErrServerClosed {
 				return
 			}
-			logger.Fatalf("rest server closed unexpect: %+v", err)
+			logger.Fatalf("rest server closed unexpect: %v", err)
 		}
 	}()
 
@@ -139,7 +139,7 @@ func (s *Server) Serve() error {
 				if err == http.ErrServerClosed {
 					return
 				}
-				logger.Fatalf("metrics server closed unexpect: %+v", err)
+				logger.Fatalf("metrics server closed unexpect: %v", err)
 			}
 		}()
 	}
@@ -147,7 +147,7 @@ func (s *Server) Serve() error {
 	// Generate GRPC listener
 	lis, _, err := rpc.ListenWithPortRange(s.config.Server.GRPC.Listen, s.config.Server.GRPC.PortRange.Start, s.config.Server.GRPC.PortRange.End)
 	if err != nil {
-		logger.Fatalf("net listener failed to start: %+v", err)
+		logger.Fatalf("net listener failed to start: %v", err)
 	}
 	defer lis.Close()
 
diff --git a/manager/service/service_grpc.go b/manager/service/service_grpc.go
index 047ecc8d..4655e8be 100644
--- a/manager/service/service_grpc.go
+++ b/manager/service/service_grpc.go
@@ -384,13 +384,13 @@ func (s *GRPC) ListSchedulers(ctx context.Context, req *manager.ListSchedulersRe
 	}
 
 	// Search optimal scheduler cluster
-	log.Infof("list scheduler clusters %+v with hostInfo %+v", schedulerClusters, req.HostInfo)
+	log.Infof("list scheduler clusters %#v with hostInfo %#v", schedulerClusters, req.HostInfo)
 	schedulerCluster, err := s.searcher.FindSchedulerCluster(ctx, schedulerClusters, req)
 	if err != nil {
 		log.Errorf("can not matching scheduler cluster %v", err)
 		return nil, status.Error(codes.NotFound, "scheduler cluster not found")
 	}
-	log.Infof("find matching scheduler cluster %v", schedulerCluster)
+	log.Infof("find matching scheduler cluster %#v", schedulerCluster)
 
 	schedulers := []model.Scheduler{}
 	if err := s.db.WithContext(ctx).Find(&schedulers, &model.Scheduler{
diff --git a/manager/types/scheduler_cluster.go b/manager/types/scheduler_cluster.go
index 0970b2a2..63069abd 100644
--- a/manager/types/scheduler_cluster.go
+++ b/manager/types/scheduler_cluster.go
@@ -59,4 +59,7 @@ type SchedulerClusterClientConfig struct {
 }
 
 type SchedulerClusterScopes struct {
+	IDC         string `yaml:"idc" mapstructure:"idc" json:"idc" binding:"omitempty"`
+	NetTopology string `yaml:"net_topology" mapstructure:"net_topology" json:"net_topology" binding:"omitempty"`
+	Location    string `yaml:"location" mapstructure:"location" json:"location" binding:"omitempty"`
 }
diff --git a/scheduler/core/events.go b/scheduler/core/events.go
index 0eed4156..e94f3917 100644
--- a/scheduler/core/events.go
+++ b/scheduler/core/events.go
@@ -372,7 +372,7 @@ func constructSuccessPeerPacket(peer *supervisor.Peer, parent *supervisor.Peer,
 		StealPeers:    stealPeers,
 		Code:          base.Code_Success,
 	}
-	logger.Debugf("send peerPacket %+v to peer %s", peerPacket, peer.ID)
+	logger.Debugf("send peerPacket %#v to peer %s", peerPacket, peer.ID)
 	return peerPacket
 }
 
diff --git a/scheduler/core/service.go b/scheduler/core/service.go
index b317151c..c4a8e780 100644
--- a/scheduler/core/service.go
+++ b/scheduler/core/service.go
@@ -209,7 +209,7 @@ func (s *SchedulerService) Stop() {
 func (s *SchedulerService) SelectParent(peer *supervisor.Peer) (parent *supervisor.Peer, err error) {
 	parent, _, hasParent := s.sched.ScheduleParent(peer, sets.NewString())
 	if !hasParent || parent == nil {
-		return nil, errors.Errorf("no parent peer available for peer %v", peer.ID)
+		return nil, errors.Errorf("no parent peer available for peer %s", peer.ID)
 	}
 	return parent, nil
 }
@@ -297,7 +297,7 @@ func (s *SchedulerService) GetOrCreateTask(ctx context.Context, task *supervisor
 			if ok = s.worker.send(peerDownloadSuccessEvent{cdnPeer, nil}); !ok {
 				logger.Error("send taskSeed success event failed, eventLoop is shutdown")
 			}
-			logger.Infof("successfully obtain seeds from cdn, task: %+v", task)
+			logger.Infof("successfully obtain seeds from cdn, task: %#v", task)
 		}
 	}()
 
diff --git a/scheduler/rpcserver/rpcserver.go b/scheduler/rpcserver/rpcserver.go
index f0a6e726..8e717ccb 100644
--- a/scheduler/rpcserver/rpcserver.go
+++ b/scheduler/rpcserver/rpcserver.go
@@ -61,7 +61,7 @@ func (s *server) RegisterPeerTask(ctx context.Context, request *scheduler.PeerTa
 	ctx, span = tracer.Start(ctx, config.SpanPeerRegister, trace.WithSpanKind(trace.SpanKindServer))
 	span.SetAttributes(config.AttributePeerRegisterRequest.String(request.String()))
 	defer span.End()
-	logger.Debugf("register peer task, req: %+v", request)
+	logger.Debugf("register peer task, req: %#v", request)
 	resp = new(scheduler.RegisterResult)
 
 	taskID := idgen.TaskID(request.Url, request.UrlMeta)
diff --git a/scheduler/scheduler.go b/scheduler/scheduler.go
index 453bdcdd..1da7df68 100644
--- a/scheduler/scheduler.go
+++ b/scheduler/scheduler.go
@@ -185,7 +185,7 @@ func (s *Server) Serve() error {
 				if err == http.ErrServerClosed {
 					return
 				}
-				logger.Fatalf("metrics server closed unexpect: %+v", err)
+				logger.Fatalf("metrics server closed unexpect: %v", err)
 			}
 		}()
 	}
@@ -205,14 +205,14 @@ func (s *Server) Serve() error {
 	// Generate GRPC listener
 	lis, _, err := rpc.ListenWithPortRange(s.config.Server.IP, s.config.Server.Port, s.config.Server.Port)
 	if err != nil {
-		logger.Fatalf("net listener failed to start: %+v", err)
+		logger.Fatalf("net listener failed to start: %v", err)
 	}
 	defer lis.Close()
 
 	// Started GRPC server
 	logger.Infof("started grpc server at %s://%s", lis.Addr().Network(), lis.Addr().String())
 	if err := s.grpcServer.Serve(lis); err != nil {
-		logger.Errorf("stoped grpc server: %+v", err)
+		logger.Errorf("stoped grpc server: %v", err)
 		return err
 	}
 
@@ -222,14 +222,14 @@ func (s *Server) Serve() error {
 func (s *Server) Stop() {
 	// Stop dynconfig server
 	if err := s.dynconfig.Stop(); err != nil {
-		logger.Errorf("dynconfig client closed failed %s", err)
+		logger.Errorf("dynconfig client closed failed %v", err)
 	}
 	logger.Info("dynconfig client closed")
 
 	// Stop manager client
 	if s.managerClient != nil {
 		if err := s.managerClient.Close(); err != nil {
-			logger.Errorf("manager client failed to stop: %+v", err)
+			logger.Errorf("manager client failed to stop: %v", err)
 		}
 		logger.Info("manager client closed")
 	}
@@ -245,7 +245,7 @@ func (s *Server) Stop() {
 	// Stop metrics server
 	if s.metricsServer != nil {
 		if err := s.metricsServer.Shutdown(context.Background()); err != nil {
-			logger.Errorf("metrics server failed to stop: %+v", err)
+			logger.Errorf("metrics server failed to stop: %v", err)
 		}
 		logger.Info("metrics server closed under request")
 	}


Test patch:
diff --git a/client/daemon/peer/peertask_file_test.go b/client/daemon/peer/peertask_file_test.go
index e748e8ab..893a6e07 100644
--- a/client/daemon/peer/peertask_file_test.go
+++ b/client/daemon/peer/peertask_file_test.go
@@ -29,10 +29,10 @@ import (
 	"github.com/golang/mock/gomock"
 	testifyassert "github.com/stretchr/testify/assert"
 
-	"d7y.io/dragonfly/v2/cdn/cdnutil"
 	"d7y.io/dragonfly/v2/client/clientutil"
 	"d7y.io/dragonfly/v2/client/config"
 	"d7y.io/dragonfly/v2/client/daemon/test"
+	"d7y.io/dragonfly/v2/internal/dfutils"
 	"d7y.io/dragonfly/v2/pkg/rpc/base"
 	"d7y.io/dragonfly/v2/pkg/rpc/scheduler"
 	"d7y.io/dragonfly/v2/pkg/source"
@@ -101,7 +101,7 @@ func TestFilePeerTask_BackSource_WithContentLength(t *testing.T) {
 		pieceManager: &pieceManager{
 			storageManager:   storageManager,
 			pieceDownloader:  downloader,
-			computePieceSize: cdnutil.ComputePieceSize,
+			computePieceSize: dfutils.ComputePieceSize,
 		},
 		storageManager:  storageManager,
 		schedulerClient: schedulerClient,
@@ -219,7 +219,7 @@ func TestFilePeerTask_BackSource_WithoutContentLength(t *testing.T) {
 		pieceManager: &pieceManager{
 			storageManager:   storageManager,
 			pieceDownloader:  downloader,
-			computePieceSize: cdnutil.ComputePieceSize,
+			computePieceSize: dfutils.ComputePieceSize,
 		},
 		storageManager:  storageManager,
 		schedulerClient: schedulerClient,
diff --git a/client/daemon/peer/peertask_manager_test.go b/client/daemon/peer/peertask_manager_test.go
index ca18b53c..1593a64e 100644
--- a/client/daemon/peer/peertask_manager_test.go
+++ b/client/daemon/peer/peertask_manager_test.go
@@ -35,13 +35,13 @@ import (
 	testifyassert "github.com/stretchr/testify/assert"
 	"google.golang.org/grpc"
 
-	"d7y.io/dragonfly/v2/cdn/cdnutil"
 	"d7y.io/dragonfly/v2/client/clientutil"
 	"d7y.io/dragonfly/v2/client/config"
 	"d7y.io/dragonfly/v2/client/daemon/storage"
 	"d7y.io/dragonfly/v2/client/daemon/test"
 	mock_daemon "d7y.io/dragonfly/v2/client/daemon/test/mock/daemon"
 	mock_scheduler "d7y.io/dragonfly/v2/client/daemon/test/mock/scheduler"
+	"d7y.io/dragonfly/v2/internal/dfutils"
 	"d7y.io/dragonfly/v2/pkg/basic/dfnet"
 	"d7y.io/dragonfly/v2/pkg/rpc"
 	"d7y.io/dragonfly/v2/pkg/rpc/base"
@@ -367,7 +367,7 @@ func TestPeerTaskManager_StartStreamPeerTask_BackSource(t *testing.T) {
 		pieceManager: &pieceManager{
 			storageManager:   storageManager,
 			pieceDownloader:  NewMockPieceDownloader(ctrl),
-			computePieceSize: cdnutil.ComputePieceSize,
+			computePieceSize: dfutils.ComputePieceSize,
 		},
 		storageManager:  storageManager,
 		schedulerClient: sched,
diff --git a/client/daemon/peer/peertask_stream_test.go b/client/daemon/peer/peertask_stream_test.go
index f8b98a22..6b50c84f 100644
--- a/client/daemon/peer/peertask_stream_test.go
+++ b/client/daemon/peer/peertask_stream_test.go
@@ -28,10 +28,10 @@ import (
 	"github.com/golang/mock/gomock"
 	testifyassert "github.com/stretchr/testify/assert"
 
-	"d7y.io/dragonfly/v2/cdn/cdnutil"
 	"d7y.io/dragonfly/v2/client/clientutil"
 	"d7y.io/dragonfly/v2/client/config"
 	"d7y.io/dragonfly/v2/client/daemon/test"
+	"d7y.io/dragonfly/v2/internal/dfutils"
 	"d7y.io/dragonfly/v2/pkg/rpc/base"
 	"d7y.io/dragonfly/v2/pkg/rpc/scheduler"
 	"d7y.io/dragonfly/v2/pkg/source"
@@ -98,7 +98,7 @@ func TestStreamPeerTask_BackSource_WithContentLength(t *testing.T) {
 		pieceManager: &pieceManager{
 			storageManager:   storageManager,
 			pieceDownloader:  downloader,
-			computePieceSize: cdnutil.ComputePieceSize,
+			computePieceSize: dfutils.ComputePieceSize,
 		},
 		storageManager:  storageManager,
 		schedulerClient: schedulerClient,
@@ -204,7 +204,7 @@ func TestStreamPeerTask_BackSource_WithoutContentLength(t *testing.T) {
 		pieceManager: &pieceManager{
 			storageManager:   storageManager,
 			pieceDownloader:  downloader,
-			computePieceSize: cdnutil.ComputePieceSize,
+			computePieceSize: dfutils.ComputePieceSize,
 		},
 		storageManager:  storageManager,
 		schedulerClient: schedulerClient,
diff --git a/internal/dfutils/dfutils_test.go b/internal/dfutils/dfutils_test.go
new file mode 100644
index 00000000..ca2401b8
--- /dev/null
+++ b/internal/dfutils/dfutils_test.go
@@ -0,0 +1,71 @@
+/*
+ *     Copyright 2020 The Dragonfly Authors
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package dfutils
+
+import (
+	"testing"
+)
+
+func TestComputePieceSize(t *testing.T) {
+	type args struct {
+		length int64
+	}
+	tests := []struct {
+		name string
+		args args
+		want uint32
+	}{
+		{
+			name: "length equal 200M and get default piece size",
+			args: args{
+				length: 200 * 1024 * 1024,
+			},
+			want: DefaultPieceSize,
+		}, {
+			name: "length smaller than 200M and get default piece size",
+			args: args{
+				length: 100 * 1024 * 1024,
+			},
+			want: DefaultPieceSize,
+		}, {
+			name: "length greater than 200M",
+			args: args{
+				length: 205 * 1024 * 1024,
+			},
+			want: DefaultPieceSize,
+		}, {
+			name: "length greater than 300M",
+			args: args{
+				length: 310 * 1024 * 1024,
+			},
+			want: DefaultPieceSize + 1*1024*1024,
+		}, {
+			name: "length reach piece size limit",
+			args: args{
+				length: 3100 * 1024 * 1024,
+			},
+			want: DefaultPieceSizeLimit,
+		},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			if got := ComputePieceSize(tt.args.length); got != tt.want {
+				t.Errorf("ComputePieceSize() = %v, want %v", got, tt.want)
+			}
+		})
+	}
+}