Repository: linkml/dm-bip
Pull request #83: Add Validation Reports
Author: login: amc-corey-cox
Created: 2025-05-15T13:54:04Z | Merged: 2025-05-16T18:19:07Z

We need some sort of output from Schema Validator to have verification that the step worked or information on the failure. This PR adds creation of the validation reports.


Diff:
diff --git a/Makefile b/Makefile
index e7fb27c..7fb07e9 100644
--- a/Makefile
+++ b/Makefile
@@ -25,6 +25,12 @@ help::
 	@echo "│     schema-clean        Remove all generated schema files   │"
 	@echo "│     schema-debug        Print all schema variables          │"
 	@echo "│                                                             │"
+	@echo "│ Validation targets:                                         │"
+	@echo "│     validate-schema     Validate generated schema           │"
+	@echo "│     validate-data       Validate data to generated schema   │"
+	@echo "│     validate-clean      Remove validation reports           │"
+	@echo "│     validate-debug      Print all validation variables      │"
+	@echo "│                                                             │"
 	@echo "│ Project targets:                                            │"
 	@echo "│     help                Print this help message             │"
 	@echo "│     all                 Install everything                  │"
diff --git a/pipeline.Makefile b/pipeline.Makefile
index a2779fb..ba525ba 100644
--- a/pipeline.Makefile
+++ b/pipeline.Makefile
@@ -6,10 +6,13 @@ DM_INPUT_DIR ?=
 DM_INPUT_FILES ?=
 DM_SCHEMA_NAME ?= Schema
 DM_OUTPUT_DIR ?= output/$(DM_SCHEMA_NAME)
+VALIDATOR_OUTPUT_DIR ?= $(DM_OUTPUT_DIR)/schema-validator-data
 
 # Derived output files
 SCHEMA_FILE := $(DM_OUTPUT_DIR)/schema-automator-data/$(DM_SCHEMA_NAME).yaml
-
+SCHEMA_LINT_LOG := $(VALIDATOR_OUTPUT_DIR)/$(DM_SCHEMA_NAME)-schema-lint.log
+SCHEMA_VALIDATE_LOG := $(VALIDATOR_OUTPUT_DIR)/$(DM_SCHEMA_NAME)-schema-validate.log
+DATA_VALIDATE_LOG := $(VALIDATOR_OUTPUT_DIR)/$(DM_SCHEMA_NAME)-data-validate.log
 
 ifdef DM_INPUT_FILES
 	INPUT_FILES := $(DM_INPUT_FILES)
@@ -97,7 +100,14 @@ schema-create: $(SCHEMA_FILE)
 
 .PHONY: schema-lint
 schema-lint: $(SCHEMA_FILE)
-	$(RUN) linkml-lint $<
+	@mkdir -p $(VALIDATOR_OUTPUT_DIR)
+	@echo "Linting schema $(SCHEMA_FILE)..."
+	@$(RUN) linkml-lint $< > $(SCHEMA_LINT_LOG) 2>&1; then \
+			echo "Schema linting passed." >> $(SCHEMA_LINT_LOG); \
+		else \
+			echo "Schema linting failed. See log for details." >> $(SCHEMA_LINT_LOG); \
+		fi; \
+	@echo "Schema linting log written to $(SCHEMA_LINT_LOG)"
 
 .PHONY: help
 .PHONY: schema-help
@@ -109,7 +119,36 @@ schema-help:
 	@:$(info =================)
 	@:$(info $(DEBUG))
 
+.PHONY: validate-schema
+validate-schema: $(SCHEMA_FILE)
+	@mkdir -p $(VALIDATOR_OUTPUT_DIR)
+	@echo "Validating schema $(SCHEMA_FILE)..."
+	@$(RUN) linkml validate --schema $< > $(SCHEMA_VALIDATE_LOG) 2>&1; then \
+			echo "  ✓ $$f passed." | tee -a $(SCHEMA_VALIDATE_LOG); \
+		else \
+			echo "  ✗ $$f failed. See $$out" | tee -a $(SCHEMA_VALIDATE_LOG); \
+		fi; \
+	@echo "Schema validation written to $(SCHEMA_VALIDATE_LOG)"
+
 
-# .PHONY: validate-schema
-# validate: $(SCHEMA_FILE)
-# 	$(RUN) linkml-validate -s $< -C $(INPUT_FILES)
+.PHONY: validate-data
+validate-data: $(SCHEMA_FILE)
+	@:$(call check_input_files)
+	@mkdir -p $(VALIDATOR_OUTPUT_DIR)
+	@rm -f $(DATA_VALIDATE_LOG)
+	@for f in $(INPUT_FILES); do \
+		class=$$(basename $$f | sed -E 's/\.[ct]sv$$//' | tr '-' '_' | tr '[:upper:]' '[:lower:]'); \
+		out="$(VALIDATOR_OUTPUT_DIR)/$$(basename $$f).validate.log"; \
+		echo "Validating $$f as class '$$class'..." | tee -a $(DATA_VALIDATE_LOG); \
+		if $(RUN) linkml validate --schema $(SCHEMA_FILE) --target-class $$class $$f 2>&1; then \
+			echo "  ✓ $$f passed." | tee -a $(DATA_VALIDATE_LOG); \
+		else \
+			echo "  ✗ $$f failed. See $$out" | tee -a $(DATA_VALIDATE_LOG); \
+		fi; \
+	done
+	@echo "Data validation summary written to $(DATA_VALIDATE_LOG)"
+
+.PHONY: validate-debug
+validate-debug:
+	@:$(info $(DEBUG))
+	
\ No newline at end of file
diff --git a/toy_data/schema-automator-data/ToySchema.yaml b/toy_data/schema-automator-data/ToySchema.yaml
index ee45423..db7b485 100644
--- a/toy_data/schema-automator-data/ToySchema.yaml
+++ b/toy_data/schema-automator-data/ToySchema.yaml
@@ -159,7 +159,7 @@ slots:
     examples:
     - value: '2024-01-21'
     from_schema: https://w3id.org/MySchema
-    range: datetime
+    range: date
   type:
     examples:
     - value: Study


Reviews:
user: login: ptgolden
body: This is great. I have a couple questions, but this is fine to merge.
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-05-16T15:55:38Z
user: login: madanucd
body: LGTM! Just a quick note — the current input specification for validation (using either DM_INPUT_FILES or DM_INPUT_DIR) could be a bit confusing for users, especially since the validation expects input files named to match schema classes. It might help to improve clarity by:

Adding clearer documentation or examples on how to specify inputs for validation.

Providing warnings or errors when no valid input files are detected.

Possibly setting a sensible default input directory to avoid having to always specify inputs.

This would enhance user experience and reduce confusion during validation runs.
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-05-16T15:56:59Z
user: login: amc-corey-cox
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-16T16:50:53Z
user: login: amc-corey-cox
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-16T16:52:01Z
user: login: amc-corey-cox
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-16T16:53:18Z
user: login: amc-corey-cox
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-16T16:55:14Z

Review comments:
diff_hunk: @@ -97,7 +100,12 @@ schema-create: $(SCHEMA_FILE)
 
 .PHONY: schema-lint
 schema-lint: $(SCHEMA_FILE)
-	$(RUN) linkml-lint $<
+	@mkdir -p $(VALIDATOR_OUTPUT_DIR)
+	@echo "Linting schema $(SCHEMA_FILE)..."
+	@$(RUN) linkml-lint $< > $(SCHEMA_LINT_LOG) 2>&1 && \
+		echo "Schema linting passed." >> $(SCHEMA_LINT_LOG) || \
path: pipeline.Makefile
user: login: ptgolden
body: This is an issue for another time, but once things are far enough along, we should talk about how and how much we want to report logs.

This (and `validate-schema`) pipe messages to a log file, whereas `validate-data` prints to stdout and `tee`s to a logfile (and also includes a little glyph to indicate success and failure). The inconsistency isn't a problem at the moment-- this is future work!
created_at: 2025-05-16T14:37:32Z
author_association: COLLABORATOR
diff_hunk: @@ -109,7 +117,34 @@ schema-help:
 	@:$(info =================)
 	@:$(info $(DEBUG))
 
+.PHONY: validate-schema
+validate-schema: $(SCHEMA_FILE)
+	@mkdir -p $(VALIDATOR_OUTPUT_DIR)
+	@echo "Validating schema $(SCHEMA_FILE)..."
+	@$(RUN) linkml validate --schema $< > $(SCHEMA_VALIDATE_LOG) 2>&1 && \
+		echo "Schema validation passed." >> $(SCHEMA_VALIDATE_LOG) || \
+		echo "Schema validation failed." >> $(SCHEMA_VALIDATE_LOG)
+	@echo "  Log written to $(SCHEMA_VALIDATE_LOG)"
+
 
-# .PHONY: validate-schema
-# validate: $(SCHEMA_FILE)
-# 	$(RUN) linkml-validate -s $< -C $(INPUT_FILES)
+.PHONY: validate-data
+validate-data: $(SCHEMA_FILE)
+	@:$(call check_input_files)
+	@mkdir -p $(VALIDATOR_OUTPUT_DIR)
+	@rm -f $(DATA_VALIDATE_LOG)
path: pipeline.Makefile
user: login: ptgolden
body: Do you think we should persist logs across different runs of validation?
created_at: 2025-05-16T14:38:29Z
author_association: COLLABORATOR
diff_hunk: @@ -109,7 +117,34 @@ schema-help:
 	@:$(info =================)
 	@:$(info $(DEBUG))
 
+.PHONY: validate-schema
+validate-schema: $(SCHEMA_FILE)
+	@mkdir -p $(VALIDATOR_OUTPUT_DIR)
+	@echo "Validating schema $(SCHEMA_FILE)..."
+	@$(RUN) linkml validate --schema $< > $(SCHEMA_VALIDATE_LOG) 2>&1 && \
+		echo "Schema validation passed." >> $(SCHEMA_VALIDATE_LOG) || \
+		echo "Schema validation failed." >> $(SCHEMA_VALIDATE_LOG)
+	@echo "  Log written to $(SCHEMA_VALIDATE_LOG)"
+
 
-# .PHONY: validate-schema
-# validate: $(SCHEMA_FILE)
-# 	$(RUN) linkml-validate -s $< -C $(INPUT_FILES)
+.PHONY: validate-data
+validate-data: $(SCHEMA_FILE)
+	@:$(call check_input_files)
+	@mkdir -p $(VALIDATOR_OUTPUT_DIR)
+	@rm -f $(DATA_VALIDATE_LOG)
+	@for f in $(INPUT_FILES); do \
+		class=$$(basename $$f | sed -E 's/\.[ct]sv$$//' | tr '-' '_' | tr '[:upper:]' '[:lower:]'); \
path: pipeline.Makefile
user: login: ptgolden
body: What would you think about having a variable in make called `GENERATED_CLASSES` that we define right after `INPUT_FILES`? It could be useful for debugging. (Or if we were to use classes in another recipe). That might look like:

```
GENERATED_CLASSES := $(shell echo $(basename $(notdir $(INPUT_FILES))) | tr '-' '_' | tr '[:upper:]' '[:lower:]')
```

(Note that this `basename` in that example is a make function that [strips suffixes](https://www.gnu.org/software/make/manual/html_node/File-Name-Functions.html#index-basename-1), rather than the coreutils program `basename` which is used in the code I'm commenting on, which doesn't).

created_at: 2025-05-16T15:17:05Z
author_association: COLLABORATOR
diff_hunk: @@ -97,7 +100,12 @@ schema-create: $(SCHEMA_FILE)
 
 .PHONY: schema-lint
 schema-lint: $(SCHEMA_FILE)
-	$(RUN) linkml-lint $<
+	@mkdir -p $(VALIDATOR_OUTPUT_DIR)
+	@echo "Linting schema $(SCHEMA_FILE)..."
+	@$(RUN) linkml-lint $< > $(SCHEMA_LINT_LOG) 2>&1 && \
path: pipeline.Makefile
user: login: ptgolden
body: Why pipe stderr to stdout rather than piping stderr (i.e. `2>`) to the log here?
created_at: 2025-05-16T15:19:39Z
author_association: COLLABORATOR
diff_hunk: @@ -109,7 +117,34 @@ schema-help:
 	@:$(info =================)
 	@:$(info $(DEBUG))
 
+.PHONY: validate-schema
+validate-schema: $(SCHEMA_FILE)
+	@mkdir -p $(VALIDATOR_OUTPUT_DIR)
+	@echo "Validating schema $(SCHEMA_FILE)..."
+	@$(RUN) linkml validate --schema $< > $(SCHEMA_VALIDATE_LOG) 2>&1 && \
+		echo "Schema validation passed." >> $(SCHEMA_VALIDATE_LOG) || \
+		echo "Schema validation failed." >> $(SCHEMA_VALIDATE_LOG)
+	@echo "  Log written to $(SCHEMA_VALIDATE_LOG)"
+
 
-# .PHONY: validate-schema
-# validate: $(SCHEMA_FILE)
-# 	$(RUN) linkml-validate -s $< -C $(INPUT_FILES)
+.PHONY: validate-data
+validate-data: $(SCHEMA_FILE)
+	@:$(call check_input_files)
+	@mkdir -p $(VALIDATOR_OUTPUT_DIR)
+	@rm -f $(DATA_VALIDATE_LOG)
+	@for f in $(INPUT_FILES); do \
+		class=$$(basename $$f | sed -E 's/\.[ct]sv$$//' | tr '-' '_' | tr '[:upper:]' '[:lower:]'); \
+		out="$(VALIDATOR_OUTPUT_DIR)/$$(basename $$f).validate.log"; \
+		echo "Validating $$f as class '$$class'..." | tee -a $(DATA_VALIDATE_LOG); \
+		if $(RUN) linkml validate --schema $(SCHEMA_FILE) --target-class $$class $$f 2>>$(DATA_VALIDATE_LOG); then \
+			echo "  ✓ $$f passed." | tee -a $(DATA_VALIDATE_LOG); \
+		else \
+			echo "  ✗ $$f failed. See $$out" | tee -a $(DATA_VALIDATE_LOG); \
+		fi; \
+	done
+	@echo "Validation summary written to $(DATA_VALIDATE_LOG)"
+
+.PHONY: validate-debug
+validate-debug:
path: pipeline.Makefile
user: login: ptgolden
body: Is this necessary, since it's the same as `schema-debug`?
created_at: 2025-05-16T15:21:27Z
author_association: COLLABORATOR
diff_hunk: @@ -109,7 +117,34 @@ schema-help:
 	@:$(info =================)
 	@:$(info $(DEBUG))
 
+.PHONY: validate-schema
+validate-schema: $(SCHEMA_FILE)
+	@mkdir -p $(VALIDATOR_OUTPUT_DIR)
+	@echo "Validating schema $(SCHEMA_FILE)..."
+	@$(RUN) linkml validate --schema $< > $(SCHEMA_VALIDATE_LOG) 2>&1 && \
+		echo "Schema validation passed." >> $(SCHEMA_VALIDATE_LOG) || \
+		echo "Schema validation failed." >> $(SCHEMA_VALIDATE_LOG)
+	@echo "  Log written to $(SCHEMA_VALIDATE_LOG)"
+
 
-# .PHONY: validate-schema
-# validate: $(SCHEMA_FILE)
-# 	$(RUN) linkml-validate -s $< -C $(INPUT_FILES)
+.PHONY: validate-data
+validate-data: $(SCHEMA_FILE)
+	@:$(call check_input_files)
+	@mkdir -p $(VALIDATOR_OUTPUT_DIR)
+	@rm -f $(DATA_VALIDATE_LOG)
+	@for f in $(INPUT_FILES); do \
+		class=$$(basename $$f | sed -E 's/\.[ct]sv$$//' | tr '-' '_' | tr '[:upper:]' '[:lower:]'); \
+		out="$(VALIDATOR_OUTPUT_DIR)/$$(basename $$f).validate.log"; \
+		echo "Validating $$f as class '$$class'..." | tee -a $(DATA_VALIDATE_LOG); \
+		if $(RUN) linkml validate --schema $(SCHEMA_FILE) --target-class $$class $$f 2>>$(DATA_VALIDATE_LOG); then \
+			echo "  ✓ $$f passed." | tee -a $(DATA_VALIDATE_LOG); \
+		else \
+			echo "  ✗ $$f failed. See $$out" | tee -a $(DATA_VALIDATE_LOG); \
+		fi; \
+	done
+	@echo "Validation summary written to $(DATA_VALIDATE_LOG)"
+
+.PHONY: validate-debug
+validate-debug:
path: pipeline.Makefile
user: login: amc-corey-cox
body: No it's not necessary. It's there as a stub in case we start  to have different validate variables. I'm actually thinking about changing all of this anyway... having a series of schema- and data-. What do you think?
created_at: 2025-05-16T16:50:53Z
author_association: COLLABORATOR
diff_hunk: @@ -97,7 +100,12 @@ schema-create: $(SCHEMA_FILE)
 
 .PHONY: schema-lint
 schema-lint: $(SCHEMA_FILE)
-	$(RUN) linkml-lint $<
+	@mkdir -p $(VALIDATOR_OUTPUT_DIR)
+	@echo "Linting schema $(SCHEMA_FILE)..."
+	@$(RUN) linkml-lint $< > $(SCHEMA_LINT_LOG) 2>&1 && \
path: pipeline.Makefile
user: login: amc-corey-cox
body: I'm used to seeing this construct but it doesn't matter to me either way. Do you have preference?
created_at: 2025-05-16T16:52:01Z
author_association: COLLABORATOR
diff_hunk: @@ -109,7 +117,34 @@ schema-help:
 	@:$(info =================)
 	@:$(info $(DEBUG))
 
+.PHONY: validate-schema
+validate-schema: $(SCHEMA_FILE)
+	@mkdir -p $(VALIDATOR_OUTPUT_DIR)
+	@echo "Validating schema $(SCHEMA_FILE)..."
+	@$(RUN) linkml validate --schema $< > $(SCHEMA_VALIDATE_LOG) 2>&1 && \
+		echo "Schema validation passed." >> $(SCHEMA_VALIDATE_LOG) || \
+		echo "Schema validation failed." >> $(SCHEMA_VALIDATE_LOG)
+	@echo "  Log written to $(SCHEMA_VALIDATE_LOG)"
+
 
-# .PHONY: validate-schema
-# validate: $(SCHEMA_FILE)
-# 	$(RUN) linkml-validate -s $< -C $(INPUT_FILES)
+.PHONY: validate-data
+validate-data: $(SCHEMA_FILE)
+	@:$(call check_input_files)
+	@mkdir -p $(VALIDATOR_OUTPUT_DIR)
+	@rm -f $(DATA_VALIDATE_LOG)
path: pipeline.Makefile
user: login: amc-corey-cox
body: Not with our current use-case. We'll be spinning up a new container each time anyway so we won't have logs persist... this is just a shortcut for development. Do you think we need this?
created_at: 2025-05-16T16:53:18Z
author_association: COLLABORATOR
diff_hunk: @@ -97,7 +100,12 @@ schema-create: $(SCHEMA_FILE)
 
 .PHONY: schema-lint
 schema-lint: $(SCHEMA_FILE)
-	$(RUN) linkml-lint $<
+	@mkdir -p $(VALIDATOR_OUTPUT_DIR)
+	@echo "Linting schema $(SCHEMA_FILE)..."
+	@$(RUN) linkml-lint $< > $(SCHEMA_LINT_LOG) 2>&1 && \
+		echo "Schema linting passed." >> $(SCHEMA_LINT_LOG) || \
path: pipeline.Makefile
user: login: amc-corey-cox
body: I think the only thing we care about is having the log at each step but the `| tee` is nice for being able to see what is going on as a developer. I think I should probably add that here.
created_at: 2025-05-16T16:55:14Z
author_association: COLLABORATOR

PR comments:
user: login: ptgolden
created_at: 2025-05-16T16:30:04Z
body: > Providing warnings or errors when no valid input files are detected.

Definitely-- this is already the case. That's what this function does:

https://github.com/linkml/dm-bip/blob/8637060047ee6e2c81a5873674840cbcd7726701/pipeline.Makefile#L62-L66

All of the recipes that depend on input files should call that before starting.

> Possibly setting a sensible default input directory to avoid having to always specify inputs.

I originally had a default input directory (called `input`) but I took it out in 98ce4c1dc56cb9ff6762c0abb6e0cfc2e913b193. My thought was that this pipeline will be deployed in sensitive contexts, and I wanted to remove the possibility of accidentally processing some files errantly.

You are right though, that we definitely need to fill out documentation. The behavior is documented in `make help`, but it should be in the documentation we generate as well.
author_association: COLLABORATOR
user: login: madanucd
created_at: 2025-05-16T16:39:19Z
body: > > Providing warnings or errors when no valid input files are detected.
> 
> Definitely-- this is already the case. That's what this function does:
> 
> https://github.com/linkml/dm-bip/blob/8637060047ee6e2c81a5873674840cbcd7726701/pipeline.Makefile#L62-L66
> 
> All of the recipes that depend on input files should call that before starting.
> 
> > Possibly setting a sensible default input directory to avoid having to always specify inputs.
> 
> I originally had a default input directory (called `input`) but I took it out in [98ce4c1](https://github.com/linkml/dm-bip/commit/98ce4c1dc56cb9ff6762c0abb6e0cfc2e913b193). My thought was that this pipeline will be deployed in sensitive contexts, and I wanted to remove the possibility of accidentally processing some files errantly.
> 
> You are right though, that we definitely need to fill out documentation. The behavior is documented in `make help`, but it should be in the documentation we generate as well.

Ah, my bad — I meant: if $(SCHEMA_FILE) is never created (e.g., due to malformed or empty input), it might be worth explicitly checking for its presence in the validate-schema and validate-data targets. That way, we can catch cases where schema generation silently fails and avoid running validations against a missing or empty file. Nothing major — happy to merge as-is.
author_association: COLLABORATOR
user: login: amc-corey-cox
created_at: 2025-05-16T17:07:26Z
body: Thanks for the input Madan. Yes, we should probably start to be more clear about how we're failing. I don't want to hold this up for that but let's try to get that into our other upcoming PRs. Thanks for the quick reviews!
author_association: COLLABORATOR