Repository: kcl-lang/kcl-go
Pull request #135 (2023-08-14 02:54:34)

feat: enhance generation of kcl schema from json schema
<!-- Thank you for contributing to KCL!

Note: 

1. With pull requests:

    - Open your pull request against "main"
    - Your pull request should have no more than two commits, if not you should squash them.
    - It should pass all tests in the available continuous integration systems such as GitHub Actions.
    - You should add/modify tests to cover your proposed code changes.
    - If your pull request contains a new feature, please document it on the README.

2. Please create an issue first to describe the problem.

    We recommend that link the issue with the PR in the following question.
    For more info, check https://kcl-lang.io/docs/community/contribute/
-->

#### 1. Does this PR affect any open issues?(Y/N) and add issue references 

- [ ] N
- [x] Y 

part of https://github.com/kcl-lang/kcl/issues/526

#### 2. What is the scope of this PR (e.g. component or file name):
pkg/tools/gen
<!-- You can add the scope of this change here. 
    e.g. 
    /src/server/core.rs,
    kcl-lang/kcl/src/parser 
-->

#### 3. Provide a description of the PR(e.g. more details, effects, motivations or doc link):

<!-- You can choose a brief description here -->
- [ ] Affects user behaviors
- [ ] Contains syntax changes
- [ ] Contains variable changes
- [ ] Contains experimental features
- [ ] Performance regression: Consumes more CPU
- [ ] Performance regression: Consumes more Memory
- [x] Other

<!-- You can add more details here.
    e.g. 
    Call method "XXXX" to ..... in order to ....,
    More details: https://XXXX.com/doc......
-->

In this PR, I make the following changes:
- fix a bug related to `enum` and `const` with type. 
```
"category": {
    "type": "string",
    "enum": [
        "A",
        "B"
    ]
}
------
// Before, we will get the following result, which accept all string.
category?: str | "A" | "B"
------
// Now, we will ignore the type for enum and const to get the correct result
category?: "A" | "B"
```

- improve the naming logic. For the root schema, it may has `$id` keyword, or `title` keyword, or neither as they are not required. So now we prefer `$id`, then `title` keyword, then name in parameter(for root schema it will be the filename). If neither exists, we will use "MyType" as default.
- handle name conflict. For the [keywords in kcl](https://kcl-lang.io/docs/next/reference/lang/tour/#keywords), if it is used as a identifier, we will add `$` before it to avoid name conflict.
- Optimize output for unsupported keywords. Previously, unsupported keywords could result in weird results and syntax errors. Now, we will convert it into "any" type to make it still work, although with some loss of information.



#### 4. Are there any breaking changes?(Y/N) and describe the breaking changes(e.g. more details, motivations or doc link):

- [x] N
- [ ] Y 

<!-- You can add more details here.
    e.g. 
    Calling method "XXXX" will cause the "XXXX", "XXXX" modules to be affected.
    More details: https://XXXX.com/doc......
-->

#### 5. Are there test cases for these changes?(Y/N) select and add more details, references or doc links:

<!-- You can choose a brief description here -->
- [x] Unit test
- [ ] Integration test
- [ ] Benchmark (add benchmark stats below)
- [ ] Manual test (add detailed scripts or steps below)
- [ ] Other

<!-- You can add more details here.
e.g. 
The test case in XXXX is used to .....
test cases in /src/tests/XXXXX
test cases https://github.com/XXX/pull/44
benchmark stats: time XXX ms
-->



Patch:
diff --git a/pkg/tools/gen/genkcl_jsonschema.go b/pkg/tools/gen/genkcl_jsonschema.go
index 8a06ab0..2f735ef 100644
--- a/pkg/tools/gen/genkcl_jsonschema.go
+++ b/pkg/tools/gen/genkcl_jsonschema.go
@@ -72,10 +72,18 @@ func convertSchemaFromJsonSchema(ctx convertContext, s *jsonschema.Schema, name
 		return convertResult{IsSchema: false}
 	}
 
-	result := convertResult{IsSchema: false, Name: name}
-	if result.Name == "" {
-		result.Name = "MyType"
+	// for the name of the result, we prefer $id, then title, then name in parameter
+	// if none of them exists, "MyType" as default
+	if id, ok := s.Keywords["$id"].(*jsonschema.ID); ok {
+		lastSlashIndex := strings.LastIndex(string(*id), "/")
+		name = strings.Replace(string(*id)[lastSlashIndex+1:], ".json", "", -1)
+	} else if title, ok := s.Keywords["title"].(*jsonschema.Title); ok {
+		name = string(*title)
+	}
+	if name == "" {
+		name = "MyType"
 	}
+	result := convertResult{IsSchema: false, Name: name}
 
 	isArray := false
 	typeList := typeUnion{}
@@ -86,11 +94,6 @@ func convertSchemaFromJsonSchema(ctx convertContext, s *jsonschema.Schema, name
 		case *jsonschema.Comment:
 		case *jsonschema.SchemaURI:
 		case *jsonschema.ID:
-			// if the schema has ID, use it as the name
-			lastSlashIndex := strings.LastIndex(string(*v), "/")
-			if lastSlashIndex != -1 {
-				result.Name = strings.Trim(string(*v)[lastSlashIndex+1:], ".json")
-			}
 		case *jsonschema.Description:
 			result.Description = string(*v)
 		case *jsonschema.Type:
@@ -145,6 +148,7 @@ func convertSchemaFromJsonSchema(ctx convertContext, s *jsonschema.Schema, name
 			result.HasDefault = true
 			result.DefaultValue = v.Data
 		case *jsonschema.Enum:
+			typeList.Items = make([]typeInterface, 0, len(*v))
 			for _, val := range *v {
 				unmarshalledVal := interface{}(nil)
 				err := json.Unmarshal(val, &unmarshalledVal)
@@ -163,17 +167,30 @@ func convertSchemaFromJsonSchema(ctx convertContext, s *jsonschema.Schema, name
 				logger.GetLogger().Warningf("failed to unmarshal const value: %s", err)
 				continue
 			}
-			typeList.Items = append(typeList.Items, typeValue{
-				Value: unmarshalledVal,
-			})
+			typeList.Items = []typeInterface{typeValue{Value: unmarshalledVal}}
 			result.HasDefault = true
 			result.DefaultValue = unmarshalledVal
 		case *jsonschema.Ref:
 			typeName := strcase.ToCamel(v.Reference[strings.LastIndex(v.Reference, "/")+1:])
-			typeList.Items = append(typeList.Items, typeCustom{Name: typeName})
+			typeList.Items = []typeInterface{typeCustom{Name: typeName}}
 		case *jsonschema.Defs:
 			for key, val := range *v {
-				ctx.resultMap[key] = convertSchemaFromJsonSchema(ctx, val, key)
+				sch := convertSchemaFromJsonSchema(ctx, val, key)
+				if !sch.IsSchema {
+					logger.GetLogger().Warningf("unsupported defining non-object: %s", key)
+					sch = convertResult{
+						IsSchema: true,
+						Name:     key,
+						schema: schema{
+							Name:              strcase.ToCamel(key),
+							HasIndexSignature: true,
+							IndexSignature: indexSignature{
+								Type: typePrimitive(typAny),
+							},
+						},
+					}
+				}
+				ctx.resultMap[key] = sch
 			}
 		case *jsonschema.AdditionalProperties:
 			switch v.SchemaType {
@@ -254,11 +271,19 @@ func convertSchemaFromJsonSchema(ctx convertContext, s *jsonschema.Schema, name
 
 	if result.IsSchema {
 		result.Type = typeCustom{Name: strcase.ToCamel(name)}
+		if len(result.Properties) == 0 && !result.HasIndexSignature {
+			result.HasIndexSignature = true
+			result.IndexSignature = indexSignature{Type: typePrimitive(typAny)}
+		}
 	} else {
-		if isArray {
-			result.Type = typeArray{Items: typeList}
+		if len(typeList.Items) != 0 {
+			if isArray {
+				result.Type = typeArray{Items: typeList}
+			} else {
+				result.Type = typeList
+			}
 		} else {
-			result.Type = typeList
+			result.Type = typePrimitive(typAny)
 		}
 	}
 	result.schema.Name = strcase.ToCamel(result.Name)
diff --git a/pkg/tools/gen/template.go b/pkg/tools/gen/template.go
index 951cfdb..b6b97b2 100644
--- a/pkg/tools/gen/template.go
+++ b/pkg/tools/gen/template.go
@@ -70,7 +70,39 @@ func formatValue(v interface{}) string {
 	}
 }
 
+var kclKeywords = map[string]struct{}{
+	"True":      {},
+	"False":     {},
+	"None":      {},
+	"Undefined": {},
+	"import":    {},
+	"and":       {},
+	"or":        {},
+	"in":        {},
+	"is":        {},
+	"not":       {},
+	"as":        {},
+	"if":        {},
+	"else":      {},
+	"elif":      {},
+	"for":       {},
+	"schema":    {},
+	"mixin":     {},
+	"protocol":  {},
+	"check":     {},
+	"assert":    {},
+	"all":       {},
+	"any":       {},
+	"map":       {},
+	"filter":    {},
+	"lambda":    {},
+	"rule":      {},
+}
+
 func formatName(name string) string {
+	if _, ok := kclKeywords[name]; ok {
+		return fmt.Sprintf("$%s", name)
+	}
 	return name
 }
 


Test patch:
diff --git a/pkg/tools/gen/testdata/jsonschema/basic/expect.k b/pkg/tools/gen/testdata/jsonschema/basic/expect.k
index dcd051c..2f38035 100644
--- a/pkg/tools/gen/testdata/jsonschema/basic/expect.k
+++ b/pkg/tools/gen/testdata/jsonschema/basic/expect.k
@@ -14,6 +14,7 @@ schema Book:
     price : float, optional
     available : bool, optional, default is True
     category : "Fiction" | "Science" | "History", optional
+    $rule : str, optional
     """
 
     title: str
@@ -21,4 +22,5 @@ schema Book:
     price?: float
     available?: bool = True
     category?: "Fiction" | "Science" | "History"
+    $rule?: str
 
diff --git a/pkg/tools/gen/testdata/jsonschema/basic/input.json b/pkg/tools/gen/testdata/jsonschema/basic/input.json
index 7b44111..f3f9f38 100644
--- a/pkg/tools/gen/testdata/jsonschema/basic/input.json
+++ b/pkg/tools/gen/testdata/jsonschema/basic/input.json
@@ -20,11 +20,15 @@
 			"default": true
 		},
 		"category": {
+			"type": "string",
 			"enum": [
 				"Fiction",
 				"Science",
 				"History"
 			]
+		},
+		"rule": {
+			"type": "string"
 		}
 	},
 	"required": [
diff --git a/pkg/tools/gen/testdata/jsonschema/unsupport/expect.k b/pkg/tools/gen/testdata/jsonschema/unsupport/expect.k
new file mode 100644
index 0000000..3a20af2
--- /dev/null
+++ b/pkg/tools/gen/testdata/jsonschema/unsupport/expect.k
@@ -0,0 +1,39 @@
+"""
+This file was generated by the KCL auto-gen tool. DO NOT EDIT.
+Editing this file might prove futile when you re-run the KCL auto-gen generate command.
+"""
+
+schema Shop:
+    """
+    Schema for representing a shop information.
+    In this test case, we use some logic keywords like "oneOf" that can't be directly converted at the moment. To make it still work, we'll convert it into "any" type.
+
+    Attributes
+    ----------
+    products : any, optional
+    """
+
+    products?: any
+
+schema Clothing:
+    """
+    Clothing
+    """
+
+    [...str]: any
+
+schema Product:
+    """
+    Product
+
+    Attributes
+    ----------
+    name : str, optional
+    price : float, optional
+    """
+
+    name?: str
+    price?: float
+
+    check:
+        price >= 0
diff --git a/pkg/tools/gen/testdata/jsonschema/unsupport/input.json b/pkg/tools/gen/testdata/jsonschema/unsupport/input.json
new file mode 100644
index 0000000..df5a5a8
--- /dev/null
+++ b/pkg/tools/gen/testdata/jsonschema/unsupport/input.json
@@ -0,0 +1,56 @@
+{
+  "$schema": "http://json-schema.org/draft-07/schema#",
+  "$id": "https://example.com/schemas/shop",
+  "description": "Schema for representing a shop information.\nIn this test case, we use some logic keywords like \"oneOf\" that can't be directly converted at the moment. To make it still work, we'll convert it into \"any\" type.",
+  "$defs": {
+    "product": {
+      "type": "object",
+      "properties": {
+        "name": {
+          "type": "string"
+        },
+        "price": {
+          "type": "number",
+          "minimum": 0
+        }
+      }
+    },
+    "clothing": {
+      "allOf": [
+        {
+          "$ref": "#/$defs/product"
+        },
+        {
+          "type": "object",
+          "properties": {
+            "type": {
+              "const": "clothing"
+            },
+            "material": {
+              "type": "string"
+            }
+          }
+        }
+      ]
+    }
+  },
+  "type": "object",
+  "properties": {
+    "products": {
+      "oneOf": [
+        {
+          "$ref": "#/$defs/clothing"
+        },
+        {
+          "type": "array",
+          "items": {
+            "$ref": "#/$defs/clothing"
+          }
+        },
+        {
+          "const": "empty"
+        }
+      ]
+    }
+  }
+}