Repository: linkml/linkml

Issue:
add link to the range of a slot in Jinja based the schema documentation
At the moment the range class of a slot is not linked to in the schema documentation. I guess, it has to do with:
https://github.com/linkml/linkml/blob/1bbf442f5c0dab5b6a4eb3309ef25b95c74d0892/linkml/generators/docgen/slot.md.jinja2#L25

where it should actually be `{{gen.link(element.range)}}`. If I'm correct, I could attempt a PR.

Comments:

Patch:
diff --git a/docs/faq/python.md b/docs/faq/python.md
new file mode 100644
index 00000000..e834ae62
--- /dev/null
+++ b/docs/faq/python.md
@@ -0,0 +1,123 @@
+# FAQ: Python
+
+this section of the FAQ is for questions about writing Python code that uses the LinkML framework.
+
+## Why would I use LinkML in my Python code?
+
+You can use LinkML without writing a line of Python code, but if you
+are coding in Python there are a number of advantages to using parts
+of the LinkML toolchain.
+
+The primary advantage is to get the benefit of a Python object model
+derived from your schema. Not only will this model gives you lots of
+benefits such as avoiding boilerplate and IDE support, you also get
+lots of "freebies" for things like loading/dumping to formats like YAML/JSON/RDF.
+
+See:
+
+- [python generator](https://linkml.io/linkml/generators/python.html)
+- [working with python](https://linkml.io/linkml/data/python.html)
+
+## Where do I find out more?
+
+- [working with python](https://linkml.io/linkml/data/python.html)
+- [developers guide](https://linkml.io/linkml/developers/index.html)
+
+## Which packages do I need to install?
+
+If you want to generate Python dataclasses, then you need the full [linkml](https://github.com/linkml/linkml) package.
+
+However, your code doesn't need this at *runtime*. Once you have generated your project files your only dependency will be on the more lightweight [linkml-runtime](https://github.com/linkml/linkml-runtime) package.
+
+*Note* you don't need to install [linkml-model](https://github.com/linkml/linkml-model), the metamodel is included as part of the runtime
+
+## How do I get started?
+
+We recommend going through the tutorial, and in particular, [Part 5: Using Python](https://linkml.io/linkml/intro/tutorial05.html)
+
+## What is a loader or a dumper?
+
+- loaders allow data to be loaded *into* python objects from supported formats
+- dumpers allow data to be loaded *from* python objects into supported formats
+
+Example dumper code:
+
+```python
+from linkml_runtime.dumpers import json_dumper
+from personinfo import Person
+
+p1 = Person(id='ORCID:9876', full_name='Lex Luthor', aliases=["Bad Guy"])
+yaml_dumper.dump(p1, to_file='my-data.yaml')
+```
+
+Then to do the reverse:
+
+```
+p1 = yaml_loader('my-data.yaml', target_class=Person)
+```
+
+In contrast to dumpers, loaders need to know the class they are loading into, as
+there is insufficient information in the json/yaml
+
+Each loader and dumper supports a pair of methods:
+
+- `load` and `loads` for loaders, for filehandles and strings respectively
+- `dump` and `dumps` for loaders, for filehandles and strings respectively
+
+This is following the convention of other python packages, such as the widely used json package
+
+Note: dumpers shouldn't be confused with *generators* - dumpers are for exporting *data* that conforms to a LinkML schema, and generators are for converting an actual LinkML schema to another framework (e.g. json-schema)
+
+## What formats are supported?
+
+The core formats for LinkML data are:
+
+* json (see json_dumper and json_loader)
+* yaml (see yaml_dumper and yaml_loader)
+* rdf (see rdflib_dumper and rdflib_loader)
+
+csv is also partially supported, but this only works for certain "shapes" of data due to the fact nesting is not possible in CSVs.
+
+Data can also be exported to and imported from SQL databases, but this is not part of the loader/dumper framework
+
+## Why do I get a "not JSON serializable" message?
+
+This is probably because you are using the `dump` method from the `json` package, which expects dictionaries, not objects.
+
+you should use `json_dumper` instead.
+
+## What is the difference between gen-python and gen-pydantic?
+
+TODO
+
+## How do I work programmatically with RDF data?
+
+TODO
+
+## How do I work programmatically with SQL databases?
+
+TODO
+
+## Is LinkML compatible with RDFLib?
+
+TODO
+
+## Is LinkML compatible with SQL Alchemy?
+
+TODO
+
+## Is LinkML compatible with FastAPI?
+
+TODO
+
+## Is LinkML compatible with Flask?
+
+TODO
+
+## Is LinkML compatible with Python GraphQL frameworks?
+
+TODO
+
+
+
+
diff --git a/linkml/generators/docgen.py b/linkml/generators/docgen.py
index 2aa37fa6..4410383f 100644
--- a/linkml/generators/docgen.py
+++ b/linkml/generators/docgen.py
@@ -57,6 +57,7 @@ class DocGenerator(Generator):
     template_mappings: Dict[str, str] = None
     directory = None
     template_directory = None
+    genmeta = False
 
     def __init__(self, schema: Union[str, TextIO, SchemaDefinition],
                  directory: str = None,
@@ -81,6 +82,7 @@ class DocGenerator(Generator):
         self.format = format
         self.directory = directory
         self.template_directory = template_directory
+        self.genmeta = genmeta
 
     def serialize(self, directory: str = None) -> None:
         """
@@ -108,6 +110,8 @@ class DocGenerator(Generator):
             self._write(out_str, directory, imported_schema.name)
         template = self._get_template('class')
         for cn, c in sv.all_classes().items():
+            if self._is_external(c):
+                continue
             n = self.name(c)
             out_str = template.render(gen=self,
                                       element=c,
@@ -115,6 +119,8 @@ class DocGenerator(Generator):
             self._write(out_str, directory, n)
         template = self._get_template('slot')
         for sn, s in sv.all_slots().items():
+            if self._is_external(s):
+                continue
             n = self.name(s)
             out_str = template.render(gen=self,
                                       element=s,
@@ -122,6 +128,8 @@ class DocGenerator(Generator):
             self._write(out_str, directory, n)
         template = self._get_template('enum')
         for en, e in sv.all_enums().items():
+            if self._is_external(e):
+                continue
             n = self.name(e)
             out_str = template.render(gen=self,
                                       element=e,
@@ -129,6 +137,8 @@ class DocGenerator(Generator):
             self._write(out_str, directory, n)
         template = self._get_template('type')
         for tn, t in sv.all_types().items():
+            if self._is_external(t):
+                continue
             n = self.name(t)
             out_str = template.render(gen=self,
                                       element=t,
@@ -192,8 +202,6 @@ class DocGenerator(Generator):
             env = Environment(loader=loader)
             return env.get_template(base_file_name)
 
-
-
     def name(self, element: Element) -> str:
         """
         Returns the name of the element in its canonical form
@@ -230,7 +238,6 @@ class DocGenerator(Generator):
         sc = element.from_schema
         return f'[{curie}]({uri})'
 
-
     def link(self, e: Union[Definition, DefinitionName]) -> str:
         """
         Render an element as a hyperlink
@@ -242,7 +249,9 @@ class DocGenerator(Generator):
             return 'NONE'
         if not isinstance(e, Definition):
             e = self.schemaview.get_element(e)
-        if isinstance(e, ClassDefinition):
+        if self._is_external(e):
+            return self.uri_link(e)
+        elif isinstance(e, ClassDefinition):
             return self._markdown_link(camelcase(e.name))
         elif isinstance(e, EnumDefinition):
             return self._markdown_link(camelcase(e.name))
@@ -253,6 +262,13 @@ class DocGenerator(Generator):
         else:
             return e.name
 
+    def _is_external(self, element: Element) -> bool:
+        # note: this is currently incomplete. See: https://github.com/linkml/linkml/issues/782
+        if element.from_schema == 'https://w3id.org/linkml/types' and not self.genmeta:
+            return True
+        else:
+            return False
+
     def _markdown_link(self, n: str, subfolder: str = None) -> str:
         if subfolder:
             rel_path = f'{subfolder}/{n}'
diff --git a/linkml/generators/docgen/common_metadata.md.jinja2 b/linkml/generators/docgen/common_metadata.md.jinja2
index e7482dd7..1f5f4e06 100644
--- a/linkml/generators/docgen/common_metadata.md.jinja2
+++ b/linkml/generators/docgen/common_metadata.md.jinja2
@@ -19,4 +19,15 @@ Instances of this class *should* have identifiers with one of the following pref
 {% for a in element.annotations -%}
 | {{a}} | {{ element.annotations[a].value }} |
 {% endfor %}
+{% endif %}
+
+{% if element.from_schema or element.imported_from %}
+### Schema Source
+
+{% if element.from_schema %}
+* from schema: {{ element.from_schema }}
+{% endif %}
+{% if element.imported_from %}
+* imported from: {{ element.imported_from }}
+{% endif %}
 {% endif %}
\ No newline at end of file
diff --git a/linkml/generators/docgen/slot.md.jinja2 b/linkml/generators/docgen/slot.md.jinja2
index 5b15e7c4..dcb16015 100644
--- a/linkml/generators/docgen/slot.md.jinja2
+++ b/linkml/generators/docgen/slot.md.jinja2
@@ -22,7 +22,7 @@ URI: [{{ gen.uri(element) }}]({{ gen.uri(element) }})
 
 ## Properties
 
- * Range: {{element.range}}
+ * Range: {{gen.link(element.range)}}
 
 {% if schemaview.usage_index().get(element.name) %}
 | used by | used in | type | used |
diff --git a/linkml/utils/converter.py b/linkml/utils/converter.py
index 8fd7a3c4..56b04f98 100644
--- a/linkml/utils/converter.py
+++ b/linkml/utils/converter.py
@@ -5,6 +5,8 @@ from typing import List
 
 import click
 from linkml_runtime.linkml_model import Prefix
+from linkml_runtime.utils import inference_utils
+from linkml_runtime.utils.inference_utils import infer_all_slot_values
 
 from linkml.utils import validation, datautils
 from linkml_runtime.utils.compile_python import compile_python
@@ -43,6 +45,10 @@ from linkml.utils.datautils import dumpers_loaders, _get_format, get_loader, _ge
               default=True,
               show_default=True,
               help="Validate against the schema")
+@click.option("--infer/--no-infer",
+              default=False,
+              show_default=True,
+              help="Infer missing slot values")
 @click.option("--context", "-c",
               multiple=True,
               help="path to JSON-LD context file. Required for RDF input/output")
@@ -50,7 +56,7 @@ from linkml.utils.datautils import dumpers_loaders, _get_format, get_loader, _ge
 def cli(input, module, target_class, context=None, output=None, input_format=None, output_format=None,
         prefix: List = [],
         target_class_from_path=None,
-        schema=None, validate=None, index_slot=None) -> None:
+        schema=None, validate=None, infer=None, index_slot=None) -> None:
     """
     Converts instance data to and from different LinkML Runtime serialization formats.
 
@@ -103,6 +109,9 @@ def cli(input, module, target_class, context=None, output=None, input_format=Non
         inargs['index_slot'] = index_slot
         inargs['schema'] = schema
     obj = loader.load(source=input,  target_class=py_target_class, **inargs)
+    if infer:
+        infer_config = inference_utils.Config(use_expressions=True, use_string_serialization=True)
+        infer_all_slot_values(obj, schemaview=sv, config=infer_config)
     if validate:
         if schema is None:
             raise Exception('--schema must be passed in order to validate. Suppress with --no-validate')


Test patch:
diff --git a/tests/test_generators/test_docgen.py b/tests/test_generators/test_docgen.py
index e771bf45..d27b506d 100644
--- a/tests/test_generators/test_docgen.py
+++ b/tests/test_generators/test_docgen.py
@@ -32,8 +32,38 @@ class DocGeneratorTestCase(unittest.TestCase):
         """ Tests basic document generator functionality  """
         gen = DocGenerator(SCHEMA, mergeimports=True, no_types_dir=True)
         md = gen.serialize(directory=MD_DIR)
+        # test class docs
         assert_mdfile_contains('Organization.md', 'Organization', after='Inheritance')
-        # TODO: add more tests
+        assert_mdfile_contains('Organization.md', '[aliases](aliases.md)', after='Slots')
+        assert_mdfile_contains('Organization.md',
+                               'URI: [ks:Organization](https://w3id.org/linkml/tests/kitchen_sink/Organization)',
+                               after='Class: Organization')
+        assert_mdfile_contains('Organization.md',
+                               'from_schema: https://w3id.org/linkml/tests/kitchen_sink',
+                               after='Class: Organization')
+        assert_mdfile_contains('Organization.md',
+                               'slot_uri: skos:altLabel',
+                               after='Induced')
+        # test type docs
+        assert_mdfile_contains('PhoneNumberType.md',
+                               'URI: http://www.w3.org/2001/XMLSchema#string',
+                               after='PhoneNumberType')
+        # test enum docs
+        assert_mdfile_contains('EmploymentEventType.md',
+                               'codes for different kinds of employment/HR related events',
+                               after='EmploymentEventType')
+        assert_mdfile_contains('EmploymentEventType.md',
+                               'PROMOTION | bizcodes:003 | promotion event',
+                               after='Permissible Values')
+        # test slot docs
+        assert_mdfile_contains('aliases.md',
+                               'http://www.w3.org/2004/02/skos/core#altLabel',
+                               after='aliases')
+        # test index docs
+        assert_mdfile_contains('aliases.md',
+                               'http://www.w3.org/2004/02/skos/core#altLabel',
+                               after='aliases')
+
 
     def test_custom_directory(self):
         """
diff --git a/tests/test_utils/input/data_example.yaml b/tests/test_utils/input/data_example.yaml
new file mode 100644
index 00000000..6cf30063
--- /dev/null
+++ b/tests/test_utils/input/data_example.yaml
@@ -0,0 +1,9 @@
+persons:
+  P:1:
+    first_name: first1
+    last_name: last1
+    age_in_years: 10
+  P:2:
+    first_name: first2
+    last_name: last2
+    age_in_months: 240
diff --git a/tests/test_utils/input/schema_with_inference.yaml b/tests/test_utils/input/schema_with_inference.yaml
new file mode 100644
index 00000000..c2751ba4
--- /dev/null
+++ b/tests/test_utils/input/schema_with_inference.yaml
@@ -0,0 +1,148 @@
+id: https://w3id.org/linkml/examples/inference
+title: inference example
+name: inference
+description: This demonstrates the use of inference
+license: https://creativecommons.org/publicdomain/zero/1.0/
+
+prefixes:
+  linkml: https://w3id.org/linkml/
+  ex: https://w3id.org/linkml/examples/inference/
+  skos: http://www.w3.org/2004/02/skos/core#
+  pav: http://purl.org/pav/
+  schema: http://schema.org/
+  sh: https://w3id.org/shacl/
+  P: http://example.org/
+
+default_prefix: ex
+default_range: string
+
+default_curi_maps:
+    - semweb_context
+
+imports:
+  - linkml:types
+
+
+#==================================
+# Slots                           #
+#==================================
+slots:
+  id:
+    identifier: true
+  name:
+    description: name
+  synonyms:
+    multivalued: true
+  full_name:
+    string_serialization: "{first_name} {last_name}"
+  first_name:
+  last_name:
+  age_in_years:
+    range: decimal
+    minimum_value: 0
+    maximum_value: 999
+    equals_expression: "{age_in_months} / 12"
+  age_in_months:
+    range: decimal
+    equals_expression: "{age_in_years} * 12"
+  is_juvenile:
+    range: boolean
+    equals_expression: "{age_in_years} < 18"
+  age_category:
+    range: AgeEnum
+    equals_expression: "case( ({age_in_years} < 2, 'infant'), ({age_in_years} < 18, 'juvenile'), ({age_in_years} > 18, 'adult') )"
+  prohibited:
+    equals_expression: "__import__('os').listdir()"
+  street:
+  city:
+  verbatim:
+  primary_address:
+    range: Address
+  description:
+  summary:
+
+    
+#==================================
+# Classes                         #
+#==================================
+
+classes:
+  Term:
+    slots:
+      - id
+      - name
+      - synonyms
+
+  Person:
+    slots:
+      - id
+      - first_name
+      - last_name
+      - full_name
+      - age_in_years
+      - age_in_months
+      - primary_address
+      - description
+      - is_juvenile
+      - age_category
+    slot_usage:
+      description:
+        string_serialization: |-
+          {last_name}, {first_name}, {primary_address}
+      summary:
+        equals_expression: |-
+          {first_name} + {last_name} + ' ' + ('AGE: '+str(age_in_years) if age_in_years else 'NO AGE SPECIFIED')
+    rules:
+      preconditions:
+        slot_conditions:
+          primary_address:
+      postconditions:
+        slot_conditions:
+          description:
+            string_serialization: |-
+              {last_name}, {first_name}, {primary_address}
+              {primary_address.street}
+              {primary_address.city}
+
+  Evil:
+    slots:
+      - prohibited
+
+  Relationship:
+    attributes:
+      person1:
+        range: Person
+        inlined: true
+      person2:
+        range: Person
+        inlined: true
+      type:
+      description:
+        string_serialization: |-
+          "{person1.last_name}, {person1.first_name}" IS {type} "{person2.last_name}, {person2.first_name}"
+      description2:
+        string_serialization: |-
+          "{person1.full_name}" IS {type} "{person2.full_name}"
+
+  Address:
+    slots:
+      - street
+      - city
+    string_serialization: |-
+      {street}
+      {city}
+
+  Container:
+    tree_root: true
+    attributes:
+      persons:
+        range: Person
+        inlined: true
+        multivalued: true
+
+enums:
+  AgeEnum:
+    permissible_values:
+      infant:
+      juvenile:
+      adult:
diff --git a/tests/test_utils/test_converter.py b/tests/test_utils/test_converter.py
new file mode 100644
index 00000000..ee9a8b16
--- /dev/null
+++ b/tests/test_utils/test_converter.py
@@ -0,0 +1,53 @@
+import json
+import unittest
+
+from click.testing import CliRunner
+from tests.test_utils.environment import env
+from linkml.utils.converter import cli
+
+SCHEMA = env.input_path('schema_with_inference.yaml')
+DATA_IN = env.input_path('data_example.yaml')
+JSON_OUT = env.expected_path('data_example.out.json')
+YAML_OUT = env.expected_path('data_example.out.yaml')
+RDF_OUT = env.expected_path('data_example.out.ttl')
+
+
+class TestCommandLineInterface(unittest.TestCase):
+
+    def setUp(self) -> None:
+        runner = CliRunner(mix_stderr=False)
+        self.runner = runner
+
+    def test_help(self):
+        result = self.runner.invoke(cli, ['--help'])
+        out = result.stdout
+        err = result.stderr
+        #print(err)
+        self.assertIn('INPUT', out)
+        #self.assertEqual(0, result.exit_code)
+
+    def test_infer_and_convert(self):
+        """
+        Tests using the --infer option to add missing values, and also roundtripping
+        through yaml->json->yaml->rdf->json
+        """
+        result = self.runner.invoke(cli, ['--infer', '-s', SCHEMA, DATA_IN, '-o', JSON_OUT])
+        result = self.runner.invoke(cli, ['-s', SCHEMA, JSON_OUT, '-t', 'yaml', '-o', YAML_OUT])
+        result = self.runner.invoke(cli, ['-s', SCHEMA, YAML_OUT, '-t', 'rdf', '-o', RDF_OUT])
+        result = self.runner.invoke(cli, ['-s', SCHEMA, RDF_OUT, '-t', 'rdf', '-o', JSON_OUT])
+        with open(JSON_OUT) as file:
+            obj = json.load(file)
+            persons = obj['persons']
+            p1 = persons['P:1']
+            p2 = persons['P:2']
+            self.assertTrue(p1['is_juvenile'])
+            self.assertTrue('is_juvenile' not in p2)
+            self.assertEqual(p1['age_in_years'], 10)
+            self.assertEqual(p1['age_in_months'], 120)
+            self.assertEqual(p1['age_category'], "juvenile")
+            self.assertEqual(p1['full_name'], "first1 last1")
+            self.assertEqual(p2['age_in_years'], 20)
+            self.assertEqual(p2['age_in_months'], 240)
+            self.assertEqual(p2['age_category'], "adult")
+            self.assertEqual(p2['full_name'], "first2 last2")
+