Repository: dragonflyoss/Dragonfly2
Pull request #1601 (2022-08-29 04:37:12)

feat: add common serialize package
Signed-off-by: Jim Ma <majinjing3@gmail.com>

<!--- Provide a general summary of your changes in the Title above -->

## Description

<!--- Describe your changes in detail -->

## Related Issue

<!--- This project only accepts pull requests related to open issues -->
<!--- If suggesting a new feature or change, please discuss it in an issue first -->
<!--- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!--- Please link to the issue here: -->

## Motivation and Context

<!--- Why is this change required? What problem does it solve? -->

## Screenshots (if appropriate)

## Types of changes

<!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->

- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
- [ ] Documentation Update (if none of the other choices apply)

## Checklist

<!--- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->

- [ ] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [ ] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.


codecov-commenter: # [Codecov](https://codecov.io/gh/dragonflyoss/Dragonfly2/pull/1601?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss) Report
> Merging [#1601](https://codecov.io/gh/dragonflyoss/Dragonfly2/pull/1601?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss) (8b19b7a) into [main](https://codecov.io/gh/dragonflyoss/Dragonfly2/commit/55f5daf25ce9c8a76f7df9970f20c04a34ca45bb?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss) (55f5daf) will **decrease** coverage by `0.23%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/dragonflyoss/Dragonfly2/pull/1601/graphs/tree.svg?width=650&height=150&src=pr&token=PUICJYSAT7&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss)](https://codecov.io/gh/dragonflyoss/Dragonfly2/pull/1601?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss)

```diff
@@            Coverage Diff             @@
##             main    #1601      +/-   ##
==========================================
- Coverage   42.46%   42.22%   -0.24%     
==========================================
  Files         120      120              
  Lines       11356    11330      -26     
==========================================
- Hits         4822     4784      -38     
- Misses       6025     6043      +18     
+ Partials      509      503       -6     
```

| Flag | Coverage Δ | |
|---|---|---|
| Object-compatibility-e2etests | `?` | |
| e2etests | `?` | |
| unittests | `42.22% <ø> (-0.24%)` | :arrow_down: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss#carryforward-flags-in-the-pull-request-comment) to find out more.

| [Impacted Files](https://codecov.io/gh/dragonflyoss/Dragonfly2/pull/1601?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss) | Coverage Δ | |
|---|---|---|
| [client/config/peerhost.go](https://codecov.io/gh/dragonflyoss/Dragonfly2/pull/1601/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss#diff-Y2xpZW50L2NvbmZpZy9wZWVyaG9zdC5nbw==) | `38.97% <ø> (+0.71%)` | :arrow_up: |
| [manager/config/config.go](https://codecov.io/gh/dragonflyoss/Dragonfly2/pull/1601/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss#diff-bWFuYWdlci9jb25maWcvY29uZmlnLmdv) | `0.00% <ø> (ø)` | |
| [client/daemon/peer/peertask\_piecetask\_poller.go](https://codecov.io/gh/dragonflyoss/Dragonfly2/pull/1601/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss#diff-Y2xpZW50L2RhZW1vbi9wZWVyL3BlZXJ0YXNrX3BpZWNldGFza19wb2xsZXIuZ28=) | `31.03% <0.00%> (-15.87%)` | :arrow_down: |
| [client/daemon/peer/peertask\_conductor.go](https://codecov.io/gh/dragonflyoss/Dragonfly2/pull/1601/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss#diff-Y2xpZW50L2RhZW1vbi9wZWVyL3BlZXJ0YXNrX2NvbmR1Y3Rvci5nbw==) | `56.10% <0.00%> (-0.67%)` | :arrow_down: |


Patch:
diff --git a/client/config/peerhost.go b/client/config/peerhost.go
index 0b4b970e..1c217634 100644
--- a/client/config/peerhost.go
+++ b/client/config/peerhost.go
@@ -28,7 +28,6 @@ import (
 	"os"
 	"path/filepath"
 	"regexp"
-	"strings"
 	"time"
 
 	"gopkg.in/yaml.v3"
@@ -40,6 +39,7 @@ import (
 	logger "d7y.io/dragonfly/v2/internal/dflog"
 	"d7y.io/dragonfly/v2/pkg/dfnet"
 	netip "d7y.io/dragonfly/v2/pkg/net/ip"
+	"d7y.io/dragonfly/v2/pkg/serialize"
 	"d7y.io/dragonfly/v2/pkg/unit"
 )
 
@@ -174,9 +174,9 @@ func ConvertPattern(p string, defaultPattern commonv1.Pattern) commonv1.Pattern
 }
 
 type GlobalSecurityOption struct {
-	AutoIssueCert bool       `mapstructure:"autoIssueCert" yaml:"autoIssueCert"`
-	CACert        PEMContent `mapstructure:"caCert" yaml:"caCert"`
-	TLSVerify     bool       `mapstructure:"tlsVerify" yaml:"tlsVerify"`
+	AutoIssueCert bool                 `mapstructure:"autoIssueCert" yaml:"autoIssueCert"`
+	CACert        serialize.PEMContent `mapstructure:"caCert" yaml:"caCert"`
+	TLSVerify     bool                 `mapstructure:"tlsVerify" yaml:"tlsVerify"`
 }
 
 type SchedulerOption struct {
@@ -527,12 +527,12 @@ type UnixListenOption struct {
 
 type SecurityOption struct {
 	// Insecure indicate enable tls or not
-	Insecure  bool        `mapstructure:"insecure" yaml:"insecure"`
-	CACert    PEMContent  `mapstructure:"caCert" yaml:"caCert"`
-	Cert      PEMContent  `mapstructure:"cert" yaml:"cert"`
-	Key       PEMContent  `mapstructure:"key" yaml:"key"`
-	TLSVerify bool        `mapstructure:"tlsVerify" yaml:"tlsVerify"`
-	TLSConfig *tls.Config `mapstructure:"tlsConfig" yaml:"tlsConfig"`
+	Insecure  bool                 `mapstructure:"insecure" yaml:"insecure"`
+	CACert    serialize.PEMContent `mapstructure:"caCert" yaml:"caCert"`
+	Cert      serialize.PEMContent `mapstructure:"cert" yaml:"cert"`
+	Key       serialize.PEMContent `mapstructure:"key" yaml:"key"`
+	TLSVerify bool                 `mapstructure:"tlsVerify" yaml:"tlsVerify"`
+	TLSConfig *tls.Config          `mapstructure:"tlsConfig" yaml:"tlsConfig"`
 }
 
 type StorageOption struct {
@@ -562,58 +562,10 @@ type ReloadOption struct {
 	Interval util.Duration `mapstructure:"interval" yaml:"interval"`
 }
 
-// PEMContent supports load PEM format from file or just inline PEM format content
-type PEMContent string
-
-func (p *PEMContent) UnmarshalJSON(b []byte) error {
-	var s string
-	err := json.Unmarshal(b, &s)
-	if err != nil {
-		return err
-	}
-
-	return p.loadPEM(s)
-}
-
-func (p *PEMContent) UnmarshalYAML(node *yaml.Node) error {
-	var s string
-	switch node.Kind {
-	case yaml.ScalarNode:
-		if err := node.Decode(&s); err != nil {
-			return err
-		}
-	default:
-		return errors.New("invalid pem content")
-	}
-
-	return p.loadPEM(s)
-}
-
-func (p *PEMContent) loadPEM(content string) error {
-	if content == "" {
-		*p = PEMContent("")
-		return nil
-	}
-	// inline PEM, just return
-	if strings.HasPrefix(strings.TrimSpace(content), "-----BEGIN ") {
-		val := strings.TrimSpace(content)
-		*p = PEMContent(val)
-		return nil
-	}
-
-	file, err := os.ReadFile(content)
-	if err != nil {
-		return err
-	}
-	val := strings.TrimSpace(string(file))
-	*p = PEMContent(val)
-	return nil
-}
-
 type tlsConfigFiles struct {
-	Cert   PEMContent `yaml:"cert" json:"cert"`
-	Key    PEMContent `yaml:"key" json:"key"`
-	CACert PEMContent `yaml:"caCert" json:"caCert"`
+	Cert   serialize.PEMContent `yaml:"cert" json:"cert"`
+	Key    serialize.PEMContent `yaml:"key" json:"key"`
+	CACert serialize.PEMContent `yaml:"caCert" json:"caCert"`
 }
 
 type TLSConfig struct {
diff --git a/cmd/dependency/dependency.go b/cmd/dependency/dependency.go
index 9488ee02..a1538cc3 100644
--- a/cmd/dependency/dependency.go
+++ b/cmd/dependency/dependency.go
@@ -50,6 +50,7 @@ import (
 	"d7y.io/dragonfly/v2/pkg/dfpath"
 	"d7y.io/dragonfly/v2/pkg/net/fqdn"
 	"d7y.io/dragonfly/v2/pkg/net/ip"
+	"d7y.io/dragonfly/v2/pkg/serialize"
 	"d7y.io/dragonfly/v2/pkg/unit"
 	"d7y.io/dragonfly/v2/version"
 )
@@ -238,7 +239,7 @@ func initDecoderConfig(dc *mapstructure.DecoderConfig) {
 			reflect.TypeOf(util.Duration{}),
 			reflect.TypeOf(&config.ProxyOption{}),
 			reflect.TypeOf(config.TCPListenPortRange{}),
-			reflect.TypeOf(config.PEMContent("")),
+			reflect.TypeOf(serialize.PEMContent("")),
 			reflect.TypeOf(config.URL{}),
 			reflect.TypeOf(config.CertPool{}),
 			reflect.TypeOf(config.Regexp{}):
diff --git a/manager/config/config.go b/manager/config/config.go
index 448f850d..4ab2155a 100644
--- a/manager/config/config.go
+++ b/manager/config/config.go
@@ -25,6 +25,7 @@ import (
 
 	"d7y.io/dragonfly/v2/cmd/dependency/base"
 	"d7y.io/dragonfly/v2/pkg/objectstorage"
+	"d7y.io/dragonfly/v2/pkg/serialize"
 )
 
 type Config struct {
@@ -250,10 +251,10 @@ type SecurityConfig struct {
 	Enable bool `yaml:"enable" mapstructure:"enable"`
 
 	// CACert is file path PEM-encoded certificate
-	CACert string `mapstructure:"caCert" yaml:"caCert"`
+	CACert serialize.PEMContent `mapstructure:"caCert" yaml:"caCert"`
 
 	// CAKey is file path of PEM-encoded private key.
-	CAKey string `mapstructure:"caKey" yaml:"caKey"`
+	CAKey serialize.PEMContent `mapstructure:"caKey" yaml:"caKey"`
 }
 
 // New config instance.
diff --git a/manager/manager.go b/manager/manager.go
index b9d1c3d9..4c166313 100644
--- a/manager/manager.go
+++ b/manager/manager.go
@@ -157,7 +157,7 @@ func New(cfg *config.Config, d dfpath.Dfpath) (*Server, error) {
 	// Initialize global certificate.
 	var options []rpcserver.Option
 	if cfg.Security.Enable {
-		cert, err := tls.LoadX509KeyPair(cfg.Security.CACert, cfg.Security.CAKey)
+		cert, err := tls.X509KeyPair([]byte(cfg.Security.CACert), []byte(cfg.Security.CAKey))
 		if err != nil {
 			return nil, err
 		}
diff --git a/pkg/serialize/pemcontent.go b/pkg/serialize/pemcontent.go
new file mode 100644
index 00000000..6c90e15e
--- /dev/null
+++ b/pkg/serialize/pemcontent.go
@@ -0,0 +1,74 @@
+/*
+ *     Copyright 2022 The Dragonfly Authors
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package serialize
+
+import (
+	"encoding/json"
+	"errors"
+	"os"
+	"strings"
+
+	"gopkg.in/yaml.v3"
+)
+
+// PEMContent supports load PEM format from file or just inline PEM format content
+type PEMContent string
+
+func (p *PEMContent) UnmarshalJSON(b []byte) error {
+	var s string
+	err := json.Unmarshal(b, &s)
+	if err != nil {
+		return err
+	}
+
+	return p.loadPEM(s)
+}
+
+func (p *PEMContent) UnmarshalYAML(node *yaml.Node) error {
+	var s string
+	switch node.Kind {
+	case yaml.ScalarNode:
+		if err := node.Decode(&s); err != nil {
+			return err
+		}
+	default:
+		return errors.New("invalid pem content")
+	}
+
+	return p.loadPEM(s)
+}
+
+func (p *PEMContent) loadPEM(content string) error {
+	if content == "" {
+		*p = PEMContent("")
+		return nil
+	}
+	// inline PEM, just return
+	if strings.HasPrefix(strings.TrimSpace(content), "-----BEGIN ") {
+		val := strings.TrimSpace(content)
+		*p = PEMContent(val)
+		return nil
+	}
+
+	file, err := os.ReadFile(content)
+	if err != nil {
+		return err
+	}
+	val := strings.TrimSpace(string(file))
+	*p = PEMContent(val)
+	return nil
+}


Test patch:
diff --git a/client/config/peerhost_test.go b/client/config/peerhost_test.go
index b77ef506..7c657e1b 100644
--- a/client/config/peerhost_test.go
+++ b/client/config/peerhost_test.go
@@ -31,6 +31,7 @@ import (
 	"d7y.io/dragonfly/v2/cmd/dependency/base"
 	"d7y.io/dragonfly/v2/manager/model"
 	"d7y.io/dragonfly/v2/pkg/dfnet"
+	"d7y.io/dragonfly/v2/pkg/serialize"
 	"d7y.io/dragonfly/v2/pkg/unit"
 )
 
@@ -225,9 +226,9 @@ func TestPeerHostOption_Load(t *testing.T) {
 	_cert, _ := os.ReadFile("./testdata/certs/sca.crt")
 	_key, _ := os.ReadFile("./testdata/certs/sca.key")
 
-	caCert := PEMContent(strings.TrimSpace(string(_caCert)))
-	cert := PEMContent(strings.TrimSpace(string(_cert)))
-	key := PEMContent(strings.TrimSpace(string(_key)))
+	caCert := serialize.PEMContent(strings.TrimSpace(string(_caCert)))
+	cert := serialize.PEMContent(strings.TrimSpace(string(_cert)))
+	key := serialize.PEMContent(strings.TrimSpace(string(_key)))
 
 	peerHostOption := &DaemonOption{
 		Options: base.Options{