Repository: apigee/registry
Pull request #452 (2022-01-20 19:09:05)

Unit tests for conformance package
* Added unit tests for `conformance-task.go` and `linter.go`
* Moved some common test helpers `InitReport()` and `InitRuleReportGroups()` to `test-helpers.go` in package `conformance`.
* I have added some comments in the PR to explain the major changes happening in the code.

TODOs to follow:
* Tests for `invokeLinter()`. https://github.com/apigee/registry/issues/451

shrutiparabgoogle: @timburks to review conformance logic and test cases.
@seaneganx for reviewing tests for best practices.
shrutiparabgoogle: > It looks like the task tests are missing a lot of non-trivial work being done because of our direct call to `computeConformanceReport` instead of testing the task using its `Run` function. Aside from the external linter problem, are there any other reasons for this approach?

The tests for `conformance-tasks` are still incomplete. If you look at the structure of `conformance-task.go`, 
- `Run()` is the main method of the command which starts execution of the task.
- Inside `Run()`, there are calls to `invokeLinter()` and `computeConformanceReport()`. This PR includes tests for `computeConformanceReport()`.
- The goal here is to add test cases for both `computeConformanceReport()` and `invokeLinter()`, which cover all the code paths in those functions (Success and Failure cases). Once we have enough coverage for both of these functions, testing `Run()` becomes simpler, we have to test `Run()` only based on what result these two functions return, tests for `Run()` will essentially cover how it is handling failures (and success) from `computeConformance()` and `invokeLinter()`.

I should have mentioned earlier that these tests are not complete yet, this PR was already quiet long, so I will be adding tests in iterations. Next PRs will include tests for `invokeLinter()` and then `Run()`.

> 
> This approach also has the cost of tightly coupling our tests with this particular implementation of the task (we can't modify `computeConformanceReport` without updating all of our tests).

Isn't that expected with unit tests? Everytime we change the logic for `computeConformanceReport()`, the tests need to be updated for new code paths. These tests are also a way of documenting code behavior, they will help understand the behavior of the code for anyone new who interacting with the codebase.
seaneganx: Thanks for the clarifications.

>tests for `Run()` will essentially cover how it is handling failures (and success) from `computeConformance()` and `invokeLinter()`.

Does this mean the `Run()` tests will be using some kind of test double to create the situations where it's handling fail/success from those private methods?

>>This approach also has the cost of tightly coupling our tests with this particular implementation of the task (we can't modify computeConformanceReport without updating all of our tests).

>Isn't that expected with unit tests? Everytime we change the logic for computeConformanceReport(), the tests need to be updated for new code paths.

I think it's expected that the interface will be tightly coupled with the tests. My concern is about tight coupling to the current implementation. Private/unexported methods like `invokeLinter` are much more likely to be replaced or removed in the future than behaviors and interface for `ComputeConformanceTask.Run`.

Generally speaking, I think this comes down to the pros and cons of testing private/unexported code. Searching online you will find mixed opinions ([Stack Overflow](https://softwareengineering.stackexchange.com/questions/100959/how-do-you-unit-test-private-methods)) about whether or not to do it. Conveniently I just realized we made the Software Engineering at Google book freely available online. See [Chapter 12](https://abseil.io/resources/swe_at_google.2.pdf#page=259) for unit testing guidance.


shrutiparabgoogle: > Thanks for the clarifications.
> 
> > tests for `Run()` will essentially cover how it is handling failures (and success) from `computeConformance()` and `invokeLinter()`.
> 
> Does this mean the `Run()` tests will be using some kind of test double to create the situations where it's handling fail/success from those private methods?

We can test `Run()` with specific inputs which will cause one of those private methods to fail/succeed.

For example, how `Run()` is tested in this PR: 
```
func (task *ExecCommandTask) Run(ctx context.Context) error {
	// The monitoring metrics/dashboards are built on top of the format of the log messages here.
	// Check the metric filters before making any changes to the format.
	// Location: registry/deployments/controller/dashboard/*
	logger := log.FromContext(ctx).WithFields(map[string]interface{}{
		"action": fmt.Sprintf("{%s}", task.Action.Command),
		"taskID": fmt.Sprintf("{%s}", task.TaskID),
	})

	err := task.executeCommand(ctx, logger)
	if err != nil {
		return err
	}

	logger.Debug("Successful Execution:")
	return nil
}
```

The `Run()` function makes a call to private method `task.executeCommand(ctx, logger)` and it is tested as follows: https://github.com/apigee/registry/pull/454/files#diff-509ea8a8192e6c1c5a56974f0e47c8bcb103951cdfca6e95d9b4d9601365dbc6R66

This is just a proposal and still open to discussion on what might be the best approach.

Patch:
diff --git a/cmd/registry/conformance/conformance-task.go b/cmd/registry/conformance/conformance-task.go
index 677cc87..b0a06dd 100644
--- a/cmd/registry/conformance/conformance-task.go
+++ b/cmd/registry/conformance/conformance-task.go
@@ -15,11 +15,14 @@
 package conformance
 
 import (
+	"bytes"
 	"context"
 	"fmt"
 	"io/ioutil"
 	"os"
+	"os/exec"
 	"path/filepath"
+	"time"
 
 	"github.com/apigee/registry/cmd/registry/core"
 	"github.com/apigee/registry/connection"
@@ -32,7 +35,7 @@ func conformanceReportName(specName, styleguideName string) string {
 	return fmt.Sprintf("%s/artifacts/conformance-%s", specName, styleguideName)
 }
 
-func initializeConformanceReport(specName string, styleguideId string) *rpc.ConformanceReport {
+func initializeConformanceReport(specName, styleguideId string) *rpc.ConformanceReport {
 	// Create an empty conformance report.
 	conformanceReport := &rpc.ConformanceReport{
 		Name:           conformanceReportName(specName, styleguideId),
@@ -107,86 +110,128 @@ func (task *ComputeConformanceTask) Run(ctx context.Context) error {
 		return err
 	}
 
-	// Generate conformance report
+	// Run the linters and compute conformance report
 	conformanceReport := initializeConformanceReport(task.Spec.GetName(), task.StyleguideId)
-	err = task.computeConformanceReport(ctx, root, conformanceReport)
-	if err != nil {
-		log.Errorf(ctx,
-			"Failed to compute the conformance for spec %s: %s",
-			task.Spec.GetName(),
-			err,
-		)
-		return err
+	guidelineReportsMap := make(map[string]int)
+	for _, metadata := range task.LintersMetadata {
+
+		linterResponse, err := task.invokeLinter(ctx, root, metadata)
+		// If a linter returned an error, we shouldn't stop linting completely across all linters and
+		// discard the conformance report for this spec. We should log but still continue, because there
+		// may still be useful information from other linters that we may be discarding.
+		if err != nil {
+			log.Errorf(ctx, "Linter error: %s", err)
+			continue
+		}
+
+		task.computeConformanceReport(ctx, conformanceReport, guidelineReportsMap, linterResponse, metadata)
 	}
 
 	return task.storeConformanceReport(ctx, conformanceReport)
 }
 
-func (task *ComputeConformanceTask) computeConformanceReport(
+func (task *ComputeConformanceTask) invokeLinter(
 	ctx context.Context,
 	specDirectory string,
+	metadata *linterMetadata) (*rpc.LinterResponse, error) {
+
+	// Formulate the request.
+	requestBytes, err := proto.Marshal(&rpc.LinterRequest{
+		SpecDirectory: specDirectory,
+		RuleIds:       metadata.rules,
+	})
+	if err != nil {
+		return nil, fmt.Errorf("Failed marshaling linterRequest, Error: %s ", err)
+	}
+
+	executableName := getLinterBinaryName(metadata.name)
+	cmd := exec.Command(executableName)
+	cmd.Stdin = bytes.NewReader(requestBytes)
+	cmd.Stderr = os.Stderr
+
+	pluginStartTime := time.Now()
+	// Run the linter.
+	output, err := cmd.Output()
+	if err != nil {
+		return nil, fmt.Errorf("Running the plugin %s return error: %s", executableName, err)
+	}
+
+	pluginElapsedTime := time.Since(pluginStartTime)
+	log.Debugf(ctx, "Plugin %s ran in time %s", executableName, pluginElapsedTime)
+
+	// Unmarshal the output bytes into a response object. If there's a failure, log and continue.
+	linterResponse := &rpc.LinterResponse{}
+	err = proto.Unmarshal(output, linterResponse)
+	if err != nil {
+		return nil, fmt.Errorf("Failed unmarshaling LinterResponse (plugins must write log messages to stderr, not stdout): %s", err)
+	}
+
+	// Check if there were any errors in the plugin.
+	if len(linterResponse.GetErrors()) > 0 {
+		return nil, fmt.Errorf("Plugin %s encountered errors: %v", executableName, linterResponse.GetErrors())
+	}
+
+	return linterResponse, nil
+}
+
+func (task *ComputeConformanceTask) computeConformanceReport(
+	ctx context.Context,
 	conformanceReport *rpc.ConformanceReport,
-) error {
+	guidelineReportsMap map[string]int,
+	linterResponse *rpc.LinterResponse,
+	linterMetadata *linterMetadata,
+) {
+
+	// Process linterResponse to generate conformance report
+	lintFiles := linterResponse.Lint.GetFiles()
+
+	for _, lintFile := range lintFiles {
+		lintProblems := lintFile.GetProblems()
+
+		// Iterate over the list of problems returned by the linter.
+		for _, problem := range lintProblems {
+			ruleMetadata, ok := linterMetadata.rulesMetadata[problem.GetRuleId()]
+			if !ok {
+				// If a problem the linter returned isn't one that we expect
+				// then we should ignore it
+				continue
+			}
 
-	guidelineReportsMap := make(map[string]*rpc.GuidelineReport)
+			guideline := ruleMetadata.guideline
+			guidelineRule := ruleMetadata.guidelineRule
 
-	// Iterate over all the linters, and lint.
-	for _, metadata := range task.LintersMetadata {
+			// Check if the guideline report for the guideline which contains this rule
+			// has already been initialized. If it hasn't then create one.
+			reportIndex, ok := guidelineReportsMap[guideline.GetId()]
+			if !ok {
+				guidelineReport := initializeGuidelineReport(guideline.GetId())
 
-		linterResponse, err := invokeLinter(ctx, specDirectory, metadata)
-		// If a linter returned an error, we shouldn't stop linting completely across all linters and
-		// discard the conformance report for this spec. We should log but still continue, because there
-		// may still be useful information from other linters that we may be discarding.
-		if err != nil {
-			log.Errorf(ctx, "Linter error: %s", err)
-			continue
-		}
+				// Create a new entry in the conformance report
+				guidelineGroup := conformanceReport.GuidelineReportGroups[guideline.GetStatus()]
+				guidelineGroup.GuidelineReports = append(guidelineGroup.GuidelineReports, guidelineReport)
+
+				// Store the index of this new entry in the map
+				reportIndex = len(guidelineGroup.GuidelineReports) - 1
+				guidelineReportsMap[guideline.GetId()] = reportIndex
+			}
 
-		// Process linterResponse to generate conformance report
-		lintFiles := linterResponse.Lint.GetFiles()
-
-		for _, lintFile := range lintFiles {
-			lintProblems := lintFile.GetProblems()
-
-			// Iterate over the list of problems returned by the linter.
-			for _, problem := range lintProblems {
-				ruleMetadata, ok := metadata.rulesMetadata[problem.GetRuleId()]
-				if !ok {
-					// If a problem the linter returned isn't one that we expect
-					// then we should ignore it
-					continue
-				}
-
-				guideline := ruleMetadata.guideline
-				guidelineRule := ruleMetadata.guidelineRule
-
-				// Check if the guideline report for the guideline which contains this rule
-				// has already been initialized. If it hasn't then create one.
-				guidelineReport, ok := guidelineReportsMap[guideline.GetId()]
-				if !ok {
-					guidelineReport = initializeGuidelineReport(guideline.GetId())
-					guidelineReportsMap[guideline.GetId()] = guidelineReport
-
-					// Create a new entry in the conformance report
-					guidelineGroup := conformanceReport.GuidelineReportGroups[guideline.Status]
-					guidelineGroup.GuidelineReports = append(guidelineGroup.GuidelineReports, guidelineReport)
-				}
-
-				ruleReport := &rpc.RuleReport{
-					RuleId:     guidelineRule.GetId(),
-					SpecName:   task.Spec.GetName(),
-					FileName:   filepath.Base(lintFile.GetFilePath()),
-					Suggestion: problem.Suggestion,
-					Location:   problem.Location,
-				}
-				// Add the rule report to the appropriate guideline report.
-				ruleGroup := guidelineReport.RuleReportGroups[guidelineRule.Severity]
-				ruleGroup.RuleReports = append(ruleGroup.RuleReports, ruleReport)
+			ruleReport := &rpc.RuleReport{
+				RuleId:     guidelineRule.GetId(),
+				SpecName:   task.Spec.GetName(),
+				FileName:   filepath.Base(lintFile.GetFilePath()),
+				Suggestion: problem.Suggestion,
+				Location:   problem.Location,
 			}
+			// Add the rule report to the appropriate guideline report.
+			guidelineGroup := conformanceReport.GuidelineReportGroups[guideline.GetStatus()]
+			if reportIndex >= len(guidelineGroup.GuidelineReports) {
+				log.Errorf(ctx, "Incorrect data in conformance report. Cannot attach entry for %s", guideline.GetId())
+				continue
+			}
+			ruleGroup := guidelineGroup.GuidelineReports[reportIndex].RuleReportGroups[guidelineRule.GetSeverity()]
+			ruleGroup.RuleReports = append(ruleGroup.RuleReports, ruleReport)
 		}
 	}
-
-	return nil
 }
 
 func (task *ComputeConformanceTask) storeConformanceReport(
diff --git a/cmd/registry/conformance/linter.go b/cmd/registry/conformance/linter.go
index 17c14f0..09fb4c5 100644
--- a/cmd/registry/conformance/linter.go
+++ b/cmd/registry/conformance/linter.go
@@ -15,16 +15,9 @@
 package conformance
 
 import (
-	"bytes"
-	"context"
 	"fmt"
-	"os"
-	"os/exec"
-	"time"
 
-	"github.com/apigee/registry/log"
 	"github.com/apigee/registry/rpc"
-	"google.golang.org/protobuf/proto"
 )
 
 type ruleMetadata struct {
@@ -53,6 +46,9 @@ func GenerateLinterMetadata(styleguide *rpc.StyleGuide) (map[string]*linterMetad
 		for _, rule := range guideline.GetRules() {
 			// Get the name of the linter associated with the rule.
 			linterName := rule.GetLinter()
+			if len(linterName) == 0 {
+				continue
+			}
 
 			metadata, ok := linterNameToMetadata[linterName]
 			if !ok {
@@ -64,14 +60,19 @@ func GenerateLinterMetadata(styleguide *rpc.StyleGuide) (map[string]*linterMetad
 				linterNameToMetadata[linterName] = metadata
 			}
 
+			linterRuleName := rule.GetLinterRulename()
+			if len(linterRuleName) == 0 {
+				continue
+			}
+
 			//Populate required metadata
-			metadata.rules = append(metadata.rules, rule.GetLinterRulename())
+			metadata.rules = append(metadata.rules, linterRuleName)
 
-			if _, ok := metadata.rulesMetadata[rule.GetLinterRulename()]; !ok {
-				metadata.rulesMetadata[rule.GetLinterRulename()] = &ruleMetadata{}
+			if _, ok := metadata.rulesMetadata[linterRuleName]; !ok {
+				metadata.rulesMetadata[linterRuleName] = &ruleMetadata{}
 			}
-			metadata.rulesMetadata[rule.GetLinterRulename()].guideline = guideline
-			metadata.rulesMetadata[rule.GetLinterRulename()].guidelineRule = rule
+			metadata.rulesMetadata[linterRuleName].guideline = guideline
+			metadata.rulesMetadata[linterRuleName].guidelineRule = rule
 		}
 	}
 
@@ -80,47 +81,3 @@ func GenerateLinterMetadata(styleguide *rpc.StyleGuide) (map[string]*linterMetad
 	}
 	return linterNameToMetadata, nil
 }
-
-func invokeLinter(
-	ctx context.Context,
-	specDirectory string,
-	metadata *linterMetadata) (*rpc.LinterResponse, error) {
-
-	// Formulate the request.
-	requestBytes, err := proto.Marshal(&rpc.LinterRequest{
-		SpecDirectory: specDirectory,
-		RuleIds:       metadata.rules,
-	})
-	if err != nil {
-		return nil, fmt.Errorf("Failed marshaling linterRequest, Error: %s ", err)
-	}
-
-	executableName := getLinterBinaryName(metadata.name)
-	cmd := exec.Command(executableName)
-	cmd.Stdin = bytes.NewReader(requestBytes)
-	cmd.Stderr = os.Stderr
-
-	pluginStartTime := time.Now()
-	// Run the linter.
-	output, err := cmd.Output()
-	if err != nil {
-		return nil, fmt.Errorf("Running the plugin %s return error: %s", executableName, err)
-	}
-
-	pluginElapsedTime := time.Since(pluginStartTime)
-	log.Debugf(ctx, "Plugin %s ran in time %s", executableName, pluginElapsedTime)
-
-	// Unmarshal the output bytes into a response object. If there's a failure, log and continue.
-	linterResponse := &rpc.LinterResponse{}
-	err = proto.Unmarshal(output, linterResponse)
-	if err != nil {
-		return nil, fmt.Errorf("Failed unmarshaling LinterResponse (plugins must write log messages to stderr, not stdout): %s", err)
-	}
-
-	// Check if there were any errors in the plugin.
-	if len(linterResponse.GetErrors()) > 0 {
-		return nil, fmt.Errorf("Plugin %s encountered errors: %v", executableName, linterResponse.GetErrors())
-	}
-
-	return linterResponse, nil
-}


Test patch:
diff --git a/cmd/registry/cmd/compute/conformance_test.go b/cmd/registry/cmd/compute/conformance_test.go
index ff510c1..db6cd08 100644
--- a/cmd/registry/cmd/compute/conformance_test.go
+++ b/cmd/registry/cmd/compute/conformance_test.go
@@ -35,60 +35,6 @@ import (
 	"google.golang.org/protobuf/testing/protocmp"
 )
 
-func initReport(t *testing.T) *rpc.ConformanceReport {
-	t.Helper()
-	return &rpc.ConformanceReport{
-		GuidelineReportGroups: []*rpc.GuidelineReportGroup{
-			{
-				Status:           rpc.Guideline_STATUS_UNSPECIFIED,
-				GuidelineReports: []*rpc.GuidelineReport{},
-			},
-			{
-				Status:           rpc.Guideline_PROPOSED,
-				GuidelineReports: []*rpc.GuidelineReport{},
-			},
-			{
-				Status:           rpc.Guideline_ACTIVE,
-				GuidelineReports: []*rpc.GuidelineReport{},
-			},
-			{
-				Status:           rpc.Guideline_DEPRECATED,
-				GuidelineReports: []*rpc.GuidelineReport{},
-			},
-			{
-				Status:           rpc.Guideline_DISABLED,
-				GuidelineReports: []*rpc.GuidelineReport{},
-			},
-		},
-	}
-}
-
-func initRuleReportGroups(t *testing.T) []*rpc.RuleReportGroup {
-	t.Helper()
-	return []*rpc.RuleReportGroup{
-		{
-			Severity:    rpc.Rule_SEVERITY_UNSPECIFIED,
-			RuleReports: []*rpc.RuleReport{},
-		},
-		{
-			Severity:    rpc.Rule_ERROR,
-			RuleReports: []*rpc.RuleReport{},
-		},
-		{
-			Severity:    rpc.Rule_WARNING,
-			RuleReports: []*rpc.RuleReport{},
-		},
-		{
-			Severity:    rpc.Rule_INFO,
-			RuleReports: []*rpc.RuleReport{},
-		},
-		{
-			Severity:    rpc.Rule_HINT,
-			RuleReports: []*rpc.RuleReport{},
-		},
-	}
-}
-
 func readAndGZipFile(t *testing.T, filename string) (*bytes.Buffer, error) {
 	t.Helper()
 	fileBytes, _ := ioutil.ReadFile(filename)
@@ -116,253 +62,283 @@ func TestConformance(t *testing.T) {
 			desc:            "normal case",
 			conformancePath: filepath.Join("testdata", "styleguide.yaml"),
 			getPattern:      "projects/conformance-test/locations/global/apis/petstore/versions/1.0.0/specs/openapi.yaml/artifacts/conformance-openapitest",
-			wantProto: func() *rpc.ConformanceReport {
-				conformance := initReport(t)
-				conformance.Name = "projects/conformance-test/locations/global/apis/petstore/versions/1.0.0/specs/openapi.yaml/artifacts/conformance-openapitest"
-				conformance.StyleguideName = "openapitest"
-
-				ruleReportGroups := initRuleReportGroups(t)
-
-				// Populate the expected severity entry
-				ruleReportGroups[rpc.Rule_ERROR].RuleReports = []*rpc.RuleReport{
-					{
-						RuleId:   "norefsiblings",
-						SpecName: "projects/conformance-test/locations/global/apis/petstore/versions/1.0.0/specs/openapi.yaml",
-					},
-				}
-
-				//Populate the expected guideline reports
-				guidelineReports := []*rpc.GuidelineReport{
+			wantProto: &rpc.ConformanceReport{
+				Name:           "projects/conformance-test/locations/global/apis/petstore/versions/1.0.0/specs/openapi.yaml/artifacts/conformance-openapitest",
+				StyleguideName: "openapitest",
+				GuidelineReportGroups: []*rpc.GuidelineReportGroup{
+					{Status: rpc.Guideline_STATUS_UNSPECIFIED},
+					{Status: rpc.Guideline_PROPOSED},
 					{
-						GuidelineId:      "refproperties",
-						RuleReportGroups: ruleReportGroups,
+						Status: rpc.Guideline_ACTIVE,
+						GuidelineReports: []*rpc.GuidelineReport{
+							{
+								GuidelineId: "refproperties",
+								RuleReportGroups: []*rpc.RuleReportGroup{
+									{Severity: rpc.Rule_SEVERITY_UNSPECIFIED},
+									{
+										Severity: rpc.Rule_ERROR,
+										RuleReports: []*rpc.RuleReport{
+											{
+												RuleId:   "norefsiblings",
+												SpecName: "projects/conformance-test/locations/global/apis/petstore/versions/1.0.0/specs/openapi.yaml",
+											},
+										},
+									},
+									{Severity: rpc.Rule_WARNING},
+									{Severity: rpc.Rule_INFO},
+									{Severity: rpc.Rule_HINT},
+								},
+							},
+						},
 					},
-				}
-
-				//Populate the expected status entry
-				conformance.GuidelineReportGroups[rpc.Guideline_ACTIVE].GuidelineReports = guidelineReports
-
-				return conformance
-			}(),
+					{Status: rpc.Guideline_DEPRECATED},
+					{Status: rpc.Guideline_DISABLED},
+				},
+			},
 		},
 		//Tests if default status and severity values are assigned properly in the absence of defined values
 		{
 			desc:            "default case",
 			conformancePath: filepath.Join("testdata", "styleguide-default.yaml"),
 			getPattern:      "projects/conformance-test/locations/global/apis/petstore/versions/1.0.0/specs/openapi.yaml/artifacts/conformance-openapitest-default",
-			wantProto: func() *rpc.ConformanceReport {
-				conformance := initReport(t)
-				conformance.Name = "projects/conformance-test/locations/global/apis/petstore/versions/1.0.0/specs/openapi.yaml/artifacts/conformance-openapitest-default"
-				conformance.StyleguideName = "openapitest-default"
-
-				ruleReportGroups := initRuleReportGroups(t)
-
-				// Populate the expected severity entry
-				ruleReportGroups[rpc.Rule_SEVERITY_UNSPECIFIED].RuleReports = []*rpc.RuleReport{
-					{
-						RuleId:   "norefsiblings",
-						SpecName: "projects/conformance-test/locations/global/apis/petstore/versions/1.0.0/specs/openapi.yaml",
-					},
-				}
-
-				//Populate the expected guideline reports
-				guidelineReports := []*rpc.GuidelineReport{
+			wantProto: &rpc.ConformanceReport{
+				Name:           "projects/conformance-test/locations/global/apis/petstore/versions/1.0.0/specs/openapi.yaml/artifacts/conformance-openapitest-default",
+				StyleguideName: "openapitest-default",
+				GuidelineReportGroups: []*rpc.GuidelineReportGroup{
 					{
-						GuidelineId:      "refproperties",
-						RuleReportGroups: ruleReportGroups,
+						Status: rpc.Guideline_STATUS_UNSPECIFIED,
+						GuidelineReports: []*rpc.GuidelineReport{
+							{
+								GuidelineId: "refproperties",
+								RuleReportGroups: []*rpc.RuleReportGroup{
+									{
+										Severity: rpc.Rule_SEVERITY_UNSPECIFIED,
+										RuleReports: []*rpc.RuleReport{
+											{
+												RuleId:   "norefsiblings",
+												SpecName: "projects/conformance-test/locations/global/apis/petstore/versions/1.0.0/specs/openapi.yaml",
+											},
+										},
+									},
+									{Severity: rpc.Rule_ERROR},
+									{Severity: rpc.Rule_WARNING},
+									{Severity: rpc.Rule_INFO},
+									{Severity: rpc.Rule_HINT},
+								},
+							},
+						},
 					},
-				}
-
-				//Populate the expected status entry
-				conformance.GuidelineReportGroups[rpc.Guideline_STATUS_UNSPECIFIED].GuidelineReports = guidelineReports
-
-				return conformance
-			}(),
+					{Status: rpc.Guideline_PROPOSED},
+					{Status: rpc.Guideline_ACTIVE},
+					{Status: rpc.Guideline_DEPRECATED},
+					{Status: rpc.Guideline_DISABLED},
+				},
+			},
 		},
 		//Tests if multiple severity levels are populated correctly in severity report
 		{
 			desc:            "multiple severity",
 			conformancePath: filepath.Join("testdata", "styleguide-multiple-severity.yaml"),
 			getPattern:      "projects/conformance-test/locations/global/apis/petstore/versions/1.0.0/specs/openapi.yaml/artifacts/conformance-openapitest-multiple-severity",
-			wantProto: func() *rpc.ConformanceReport {
-				conformance := initReport(t)
-				conformance.Name = "projects/conformance-test/locations/global/apis/petstore/versions/1.0.0/specs/openapi.yaml/artifacts/conformance-openapitest-multiple-severity"
-				conformance.StyleguideName = "openapitest-multiple-severity"
-
-				ruleReportGroups := initRuleReportGroups(t)
-
-				// Populate the expected severity entry
-				ruleReportGroups[rpc.Rule_INFO].RuleReports = []*rpc.RuleReport{
-					{
-						RuleId:   "openapitags",
-						SpecName: "projects/conformance-test/locations/global/apis/petstore/versions/1.0.0/specs/openapi.yaml",
-					},
-					{
-						RuleId:   "openapitagsalphabetical",
-						SpecName: "projects/conformance-test/locations/global/apis/petstore/versions/1.0.0/specs/openapi.yaml",
-					},
-				}
-				ruleReportGroups[rpc.Rule_ERROR].RuleReports = []*rpc.RuleReport{
+			wantProto: &rpc.ConformanceReport{
+				Name:           "projects/conformance-test/locations/global/apis/petstore/versions/1.0.0/specs/openapi.yaml/artifacts/conformance-openapitest-multiple-severity",
+				StyleguideName: "openapitest-multiple-severity",
+				GuidelineReportGroups: []*rpc.GuidelineReportGroup{
+					{Status: rpc.Guideline_STATUS_UNSPECIFIED},
+					{Status: rpc.Guideline_PROPOSED},
 					{
-						RuleId:   "operationtags",
-						SpecName: "projects/conformance-test/locations/global/apis/petstore/versions/1.0.0/specs/openapi.yaml",
+						Status: rpc.Guideline_ACTIVE,
+						GuidelineReports: []*rpc.GuidelineReport{
+							{
+								GuidelineId: "tagproperties",
+								RuleReportGroups: []*rpc.RuleReportGroup{
+									{Severity: rpc.Rule_SEVERITY_UNSPECIFIED},
+									{
+										Severity: rpc.Rule_ERROR,
+										RuleReports: []*rpc.RuleReport{
+											{
+												RuleId:   "operationtags",
+												SpecName: "projects/conformance-test/locations/global/apis/petstore/versions/1.0.0/specs/openapi.yaml",
+											},
+											{
+												RuleId:   "operationtagdefined",
+												SpecName: "projects/conformance-test/locations/global/apis/petstore/versions/1.0.0/specs/openapi.yaml",
+											},
+										},
+									},
+									{Severity: rpc.Rule_WARNING},
+									{
+										Severity: rpc.Rule_INFO,
+										RuleReports: []*rpc.RuleReport{
+											{
+												RuleId:   "openapitags",
+												SpecName: "projects/conformance-test/locations/global/apis/petstore/versions/1.0.0/specs/openapi.yaml",
+											},
+											{
+												RuleId:   "openapitagsalphabetical",
+												SpecName: "projects/conformance-test/locations/global/apis/petstore/versions/1.0.0/specs/openapi.yaml",
+											},
+										},
+									},
+									{Severity: rpc.Rule_HINT},
+								},
+							},
+						},
 					},
-					{
-						RuleId:   "operationtagdefined",
-						SpecName: "projects/conformance-test/locations/global/apis/petstore/versions/1.0.0/specs/openapi.yaml",
-					},
-				}
-
-				//Populate the expected guideline reports
-				guidelineReports := []*rpc.GuidelineReport{
-					{
-						GuidelineId:      "tagproperties",
-						RuleReportGroups: ruleReportGroups,
-					},
-				}
-
-				//Populate the expected status entry
-				conformance.GuidelineReportGroups[rpc.Guideline_ACTIVE].GuidelineReports = guidelineReports
-
-				return conformance
-			}(),
+					{Status: rpc.Guideline_DEPRECATED},
+					{Status: rpc.Guideline_DISABLED},
+				},
+			},
 		},
 		//Tests if multiple status entries are populated correctly in severity report
 		{
 			desc:            "multiple status",
 			conformancePath: filepath.Join("testdata", "styleguide-multiple-status.yaml"),
 			getPattern:      "projects/conformance-test/locations/global/apis/petstore/versions/1.0.0/specs/openapi.yaml/artifacts/conformance-openapitest-multiple-status",
-			wantProto: func() *rpc.ConformanceReport {
-				conformance := initReport(t)
-				conformance.Name = "projects/conformance-test/locations/global/apis/petstore/versions/1.0.0/specs/openapi.yaml/artifacts/conformance-openapitest-multiple-status"
-				conformance.StyleguideName = "openapitest-multiple-status"
-
-				// EXPECTED RULE REPORTS FOR STATUS = "ACTIVE"
-				ruleReportGroups := initRuleReportGroups(t)
-
-				// Populate the expected severity entry
-				ruleReportGroups[rpc.Rule_ERROR].RuleReports = []*rpc.RuleReport{
-					{
-						RuleId:   "norefsiblings",
-						SpecName: "projects/conformance-test/locations/global/apis/petstore/versions/1.0.0/specs/openapi.yaml",
-					},
-				}
-
-				//Populate the expected guideline reports
-				guidelineReports := []*rpc.GuidelineReport{
-					{
-						GuidelineId:      "refproperties",
-						RuleReportGroups: ruleReportGroups,
-					},
-				}
-
-				//Populate the expected status entry
-				conformance.GuidelineReportGroups[rpc.Guideline_ACTIVE].GuidelineReports = guidelineReports
-
-				//EXPECTED RULE REPORTS FOR STATUS = "PROPOSED"
-				ruleReportGroups = initRuleReportGroups(t)
-
-				// Populate the expected severity entry
-				ruleReportGroups[rpc.Rule_INFO].RuleReports = []*rpc.RuleReport{
-					{
-						RuleId:   "openapitags",
-						SpecName: "projects/conformance-test/locations/global/apis/petstore/versions/1.0.0/specs/openapi.yaml",
-					},
-				}
-				ruleReportGroups[rpc.Rule_ERROR].RuleReports = []*rpc.RuleReport{
+			wantProto: &rpc.ConformanceReport{
+				Name:           "projects/conformance-test/locations/global/apis/petstore/versions/1.0.0/specs/openapi.yaml/artifacts/conformance-openapitest-multiple-status",
+				StyleguideName: "openapitest-multiple-status",
+				GuidelineReportGroups: []*rpc.GuidelineReportGroup{
+					{Status: rpc.Guideline_STATUS_UNSPECIFIED},
 					{
-						RuleId:   "operationtags",
-						SpecName: "projects/conformance-test/locations/global/apis/petstore/versions/1.0.0/specs/openapi.yaml",
+						Status: rpc.Guideline_PROPOSED,
+						GuidelineReports: []*rpc.GuidelineReport{
+							{
+								GuidelineId: "tagproperties",
+								RuleReportGroups: []*rpc.RuleReportGroup{
+									{Severity: rpc.Rule_SEVERITY_UNSPECIFIED},
+									{
+										Severity: rpc.Rule_ERROR,
+										RuleReports: []*rpc.RuleReport{
+											{
+												RuleId:   "operationtags",
+												SpecName: "projects/conformance-test/locations/global/apis/petstore/versions/1.0.0/specs/openapi.yaml",
+											},
+										},
+									},
+									{Severity: rpc.Rule_WARNING},
+									{
+										Severity: rpc.Rule_INFO,
+										RuleReports: []*rpc.RuleReport{
+											{
+												RuleId:   "openapitags",
+												SpecName: "projects/conformance-test/locations/global/apis/petstore/versions/1.0.0/specs/openapi.yaml",
+											},
+										},
+									},
+									{Severity: rpc.Rule_HINT},
+								},
+							},
+						},
 					},
-				}
-
-				//Populate the expected guideline reports
-				guidelineReports = []*rpc.GuidelineReport{
 					{
-						GuidelineId:      "tagproperties",
-						RuleReportGroups: ruleReportGroups,
+						Status: rpc.Guideline_ACTIVE,
+						GuidelineReports: []*rpc.GuidelineReport{
+							{
+								GuidelineId: "refproperties",
+								RuleReportGroups: []*rpc.RuleReportGroup{
+									{Severity: rpc.Rule_SEVERITY_UNSPECIFIED},
+									{
+										Severity: rpc.Rule_ERROR,
+										RuleReports: []*rpc.RuleReport{
+											{
+												RuleId:   "norefsiblings",
+												SpecName: "projects/conformance-test/locations/global/apis/petstore/versions/1.0.0/specs/openapi.yaml",
+											},
+										},
+									},
+									{Severity: rpc.Rule_WARNING},
+									{Severity: rpc.Rule_INFO},
+									{Severity: rpc.Rule_HINT},
+								},
+							},
+						},
 					},
-				}
-
-				//Populate the expected status entry
-				conformance.GuidelineReportGroups[rpc.Guideline_PROPOSED].GuidelineReports = guidelineReports
-
-				return conformance
-			}(),
+					{Status: rpc.Guideline_DEPRECATED},
+					{Status: rpc.Guideline_DISABLED},
+				},
+			},
 		},
 		//Tests a guideline which defines rules from multiple linters
 		{
 			desc:            "multiple linter",
 			conformancePath: filepath.Join("testdata", "styleguide-multiple-linter.yaml"),
 			getPattern:      "projects/conformance-test/locations/global/apis/petstore/versions/1.0.0/specs/openapi.yaml/artifacts/conformance-openapitest-multiple-linter",
-			wantProto: func() *rpc.ConformanceReport {
-				conformance := initReport(t)
-				conformance.Name = "projects/conformance-test/locations/global/apis/petstore/versions/1.0.0/specs/openapi.yaml/artifacts/conformance-openapitest-multiple-linter"
-				conformance.StyleguideName = "openapitest-multiple-linter"
-
-				// EXPECTED RULE REPORTS FOR STATUS = "ACTIVE"
-				ruleReportGroups := initRuleReportGroups(t)
-
-				// Populate the expected severity entry
-				ruleReportGroups[rpc.Rule_ERROR].RuleReports = []*rpc.RuleReport{
-					{
-						RuleId:   "norefsiblings",
-						SpecName: "projects/conformance-test/locations/global/apis/petstore/versions/1.0.0/specs/openapi.yaml",
-					},
-				}
-
-				//Populate the expected guideline reports
-				guidelineReports := []*rpc.GuidelineReport{
-					{
-						GuidelineId:      "refproperties",
-						RuleReportGroups: ruleReportGroups,
-					},
-				}
-
-				//Populate the expected status entry
-				conformance.GuidelineReportGroups[rpc.Guideline_ACTIVE].GuidelineReports = guidelineReports
-
-				//EXPECTED RULE REPORTS FOR STATUS = "PROPOSED"
-				ruleReportGroups = initRuleReportGroups(t)
-
-				// Populate the expected severity entry
-				ruleReportGroups[rpc.Rule_INFO].RuleReports = []*rpc.RuleReport{
+			wantProto: &rpc.ConformanceReport{
+				Name:           "projects/conformance-test/locations/global/apis/petstore/versions/1.0.0/specs/openapi.yaml/artifacts/conformance-openapitest-multiple-linter",
+				StyleguideName: "openapitest-multiple-linter",
+				GuidelineReportGroups: []*rpc.GuidelineReportGroup{
+					{Status: rpc.Guideline_STATUS_UNSPECIFIED},
 					{
-						RuleId:   "tagdescription",
-						SpecName: "projects/conformance-test/locations/global/apis/petstore/versions/1.0.0/specs/openapi.yaml",
+						Status: rpc.Guideline_PROPOSED,
+						GuidelineReports: []*rpc.GuidelineReport{
+							{
+								GuidelineId: "descriptionproperties",
+								RuleReportGroups: []*rpc.RuleReportGroup{
+									{Severity: rpc.Rule_SEVERITY_UNSPECIFIED},
+									{
+										Severity: rpc.Rule_ERROR,
+										RuleReports: []*rpc.RuleReport{
+											{
+												RuleId:   "operationdescription",
+												SpecName: "projects/conformance-test/locations/global/apis/petstore/versions/1.0.0/specs/openapi.yaml",
+											},
+											{
+												RuleId:   "infodescription",
+												SpecName: "projects/conformance-test/locations/global/apis/petstore/versions/1.0.0/specs/openapi.yaml",
+											},
+										},
+									},
+									{
+										Severity: rpc.Rule_WARNING,
+										RuleReports: []*rpc.RuleReport{
+											{
+												RuleId:   "descriptiontags",
+												SpecName: "projects/conformance-test/locations/global/apis/petstore/versions/1.0.0/specs/openapi.yaml",
+											},
+										},
+									},
+									{
+										Severity: rpc.Rule_INFO,
+										RuleReports: []*rpc.RuleReport{
+											{
+												RuleId:   "tagdescription",
+												SpecName: "projects/conformance-test/locations/global/apis/petstore/versions/1.0.0/specs/openapi.yaml",
+											},
+										},
+									},
+									{Severity: rpc.Rule_HINT},
+								},
+							},
+						},
 					},
-				}
-				ruleReportGroups[rpc.Rule_ERROR].RuleReports = []*rpc.RuleReport{
 					{
-						RuleId:   "operationdescription",
-						SpecName: "projects/conformance-test/locations/global/apis/petstore/versions/1.0.0/specs/openapi.yaml",
+						Status: rpc.Guideline_ACTIVE,
+						GuidelineReports: []*rpc.GuidelineReport{
+							{
+								GuidelineId: "refproperties",
+								RuleReportGroups: []*rpc.RuleReportGroup{
+									{Severity: rpc.Rule_SEVERITY_UNSPECIFIED},
+									{
+										Severity: rpc.Rule_ERROR,
+										RuleReports: []*rpc.RuleReport{
+											{
+												RuleId:   "norefsiblings",
+												SpecName: "projects/conformance-test/locations/global/apis/petstore/versions/1.0.0/specs/openapi.yaml",
+											},
+										},
+									},
+									{Severity: rpc.Rule_WARNING},
+									{Severity: rpc.Rule_INFO},
+									{Severity: rpc.Rule_HINT},
+								},
+							},
+						},
 					},
-					{
-						RuleId:   "infodescription",
-						SpecName: "projects/conformance-test/locations/global/apis/petstore/versions/1.0.0/specs/openapi.yaml",
-					},
-				}
-
-				ruleReportGroups[rpc.Rule_WARNING].RuleReports = []*rpc.RuleReport{
-					{
-						RuleId:   "descriptiontags",
-						SpecName: "projects/conformance-test/locations/global/apis/petstore/versions/1.0.0/specs/openapi.yaml",
-					},
-				}
-
-				//Populate the expected guideline reports
-				guidelineReports = []*rpc.GuidelineReport{
-					{
-						GuidelineId:      "descriptionproperties",
-						RuleReportGroups: ruleReportGroups,
-					},
-				}
-
-				//Populate the expected status entry
-				conformance.GuidelineReportGroups[rpc.Guideline_PROPOSED].GuidelineReports = guidelineReports
-
-				return conformance
-			}(),
+					{Status: rpc.Guideline_DEPRECATED},
+					{Status: rpc.Guideline_DISABLED},
+				},
+			},
 		},
 	}
 
diff --git a/cmd/registry/conformance/conformance-task_test.go b/cmd/registry/conformance/conformance-task_test.go
new file mode 100644
index 0000000..c22fc14
--- /dev/null
+++ b/cmd/registry/conformance/conformance-task_test.go
@@ -0,0 +1,598 @@
+// Copyright 2022 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package conformance
+
+import (
+	"context"
+	"fmt"
+	"testing"
+
+	"github.com/apigee/registry/rpc"
+	"github.com/google/go-cmp/cmp"
+	"github.com/google/go-cmp/cmp/cmpopts"
+	"google.golang.org/protobuf/testing/protocmp"
+)
+
+const specName = "projects/demo/locations/global/apis/petstore/versions/1.0.0/specs/openapi.yaml"
+const styleguideId = "openapi-test"
+
+// This test will catch any changes made to the original status values.
+func TestInitializeConformanceReport(t *testing.T) {
+	want := &rpc.ConformanceReport{
+		Name:           fmt.Sprintf("%s/artifacts/conformance-%s", specName, styleguideId),
+		StyleguideName: styleguideId,
+		GuidelineReportGroups: []*rpc.GuidelineReportGroup{
+			{
+				Status:           rpc.Guideline_STATUS_UNSPECIFIED,
+				GuidelineReports: make([]*rpc.GuidelineReport, 0),
+			},
+			{
+				Status:           rpc.Guideline_PROPOSED,
+				GuidelineReports: make([]*rpc.GuidelineReport, 0),
+			},
+			{
+				Status:           rpc.Guideline_ACTIVE,
+				GuidelineReports: make([]*rpc.GuidelineReport, 0),
+			},
+			{
+				Status:           rpc.Guideline_DEPRECATED,
+				GuidelineReports: make([]*rpc.GuidelineReport, 0),
+			},
+			{
+				Status:           rpc.Guideline_DISABLED,
+				GuidelineReports: make([]*rpc.GuidelineReport, 0),
+			},
+		},
+	}
+
+	got := initializeConformanceReport(specName, styleguideId)
+	opts := cmp.Options{
+		protocmp.Transform(),
+		cmpopts.SortSlices(func(a, b string) bool { return a < b }),
+	}
+	if !cmp.Equal(want, got, opts) {
+		t.Errorf("GetDiff returned unexpected diff (-want +got):\n%s", cmp.Diff(want, got, opts))
+	}
+}
+
+// This test will catch any changes made to the original severity values.
+func TestInitializeGuidelineReport(t *testing.T) {
+	want := &rpc.GuidelineReport{
+		GuidelineId: styleguideId,
+		RuleReportGroups: []*rpc.RuleReportGroup{
+			{
+				Severity:    rpc.Rule_SEVERITY_UNSPECIFIED,
+				RuleReports: make([]*rpc.RuleReport, 0),
+			},
+			{
+				Severity:    rpc.Rule_ERROR,
+				RuleReports: make([]*rpc.RuleReport, 0),
+			},
+			{
+				Severity:    rpc.Rule_WARNING,
+				RuleReports: make([]*rpc.RuleReport, 0),
+			},
+			{
+				Severity:    rpc.Rule_INFO,
+				RuleReports: make([]*rpc.RuleReport, 0),
+			},
+			{
+				Severity:    rpc.Rule_HINT,
+				RuleReports: make([]*rpc.RuleReport, 0),
+			},
+		},
+	}
+
+	got := initializeGuidelineReport(styleguideId)
+	opts := cmp.Options{
+		protocmp.Transform(),
+		cmpopts.SortSlices(func(a, b string) bool { return a < b }),
+	}
+	if !cmp.Equal(want, got, opts) {
+		t.Errorf("GetDiff returned unexpected diff (-want +got):\n%s", cmp.Diff(want, got, opts))
+	}
+}
+
+func TestComputeConformanceReport(t *testing.T) {
+	tests := []struct {
+		desc           string
+		linterResponse *rpc.LinterResponse
+		linterMetadata *linterMetadata
+		wantReport     *rpc.ConformanceReport
+	}{
+		// Test basic flow.
+		{
+			desc: "Normal case",
+			linterResponse: &rpc.LinterResponse{
+				Lint: &rpc.Lint{
+					Name: "sample-result",
+					Files: []*rpc.LintFile{
+						{
+							FilePath: "test-result-file",
+							Problems: []*rpc.LintProblem{
+								{
+									Message:    "no-$ref-siblings violated",
+									RuleId:     "no-$ref-siblings",
+									Suggestion: "fix no-$ref-siblings",
+									Location: &rpc.LintLocation{
+										StartPosition: &rpc.LintPosition{LineNumber: 11, ColumnNumber: 25},
+										EndPosition:   &rpc.LintPosition{LineNumber: 11, ColumnNumber: 32},
+									},
+								},
+							},
+						},
+					},
+				},
+			},
+			linterMetadata: &linterMetadata{
+				name:  "sample-linter",
+				rules: []string{"no-$ref-siblings"},
+				rulesMetadata: map[string]*ruleMetadata{
+					"no-$ref-siblings": {
+						guidelineRule: &rpc.Rule{
+							Id:       "norefsiblings",
+							Severity: rpc.Rule_ERROR,
+						},
+						guideline: &rpc.Guideline{
+							Id:     "refproperties",
+							Status: rpc.Guideline_ACTIVE,
+						},
+					},
+				},
+			},
+			wantReport: &rpc.ConformanceReport{
+				Name:           fmt.Sprintf("%s/artifacts/conformance-%s", specName, styleguideId),
+				StyleguideName: styleguideId,
+				GuidelineReportGroups: []*rpc.GuidelineReportGroup{
+					{Status: rpc.Guideline_STATUS_UNSPECIFIED},
+					{Status: rpc.Guideline_PROPOSED},
+					{
+						Status: rpc.Guideline_ACTIVE,
+						GuidelineReports: []*rpc.GuidelineReport{
+							{
+								GuidelineId: "refproperties",
+								RuleReportGroups: []*rpc.RuleReportGroup{
+									{Severity: rpc.Rule_SEVERITY_UNSPECIFIED},
+									{
+										Severity: rpc.Rule_ERROR,
+										RuleReports: []*rpc.RuleReport{
+											{
+												RuleId:     "norefsiblings",
+												SpecName:   specName,
+												FileName:   "test-result-file",
+												Suggestion: "fix no-$ref-siblings",
+												Location: &rpc.LintLocation{
+													StartPosition: &rpc.LintPosition{LineNumber: 11, ColumnNumber: 25},
+													EndPosition:   &rpc.LintPosition{LineNumber: 11, ColumnNumber: 32},
+												},
+											},
+										},
+									},
+									{Severity: rpc.Rule_WARNING},
+									{Severity: rpc.Rule_INFO},
+									{Severity: rpc.Rule_HINT},
+								},
+							},
+						},
+					},
+					{Status: rpc.Guideline_DEPRECATED},
+					{Status: rpc.Guideline_DISABLED},
+				},
+			},
+		},
+		// Test: LinterResponse includes multiple rule violations. Validate that only the once configured in the styleguide show up in the final conformance report.
+		{
+			desc: "Multiple violations",
+			linterResponse: &rpc.LinterResponse{
+				Lint: &rpc.Lint{
+					Name: "sample-result",
+					Files: []*rpc.LintFile{
+						{
+							FilePath: "test-result-file",
+							Problems: []*rpc.LintProblem{
+								{
+									Message:    "tag-description violated",
+									RuleId:     "tag-description",
+									Suggestion: "fix tag-description",
+								},
+								{
+									Message:    "operation-description violated",
+									RuleId:     "operation-description",
+									Suggestion: "fix operation-description",
+								},
+							},
+						},
+					},
+				},
+			},
+			linterMetadata: &linterMetadata{
+				name:  "sample-linter",
+				rules: []string{"operation-description"},
+				rulesMetadata: map[string]*ruleMetadata{
+					"operation-description": {
+						guidelineRule: &rpc.Rule{
+							Id:       "operationdescription",
+							Severity: rpc.Rule_ERROR,
+						},
+						guideline: &rpc.Guideline{
+							Id:     "descriptionproperties",
+							Status: rpc.Guideline_ACTIVE,
+						},
+					},
+				},
+			},
+			wantReport: &rpc.ConformanceReport{
+				Name:           fmt.Sprintf("%s/artifacts/conformance-%s", specName, styleguideId),
+				StyleguideName: styleguideId,
+				GuidelineReportGroups: []*rpc.GuidelineReportGroup{
+					{Status: rpc.Guideline_STATUS_UNSPECIFIED},
+					{Status: rpc.Guideline_PROPOSED},
+					{
+						Status: rpc.Guideline_ACTIVE,
+						GuidelineReports: []*rpc.GuidelineReport{
+							{
+								GuidelineId: "descriptionproperties",
+								RuleReportGroups: []*rpc.RuleReportGroup{
+									{Severity: rpc.Rule_SEVERITY_UNSPECIFIED},
+									{
+										Severity: rpc.Rule_ERROR,
+										RuleReports: []*rpc.RuleReport{
+											{
+												RuleId:     "operationdescription",
+												SpecName:   specName,
+												FileName:   "test-result-file",
+												Suggestion: "fix operation-description",
+											},
+										},
+									},
+									{Severity: rpc.Rule_WARNING},
+									{Severity: rpc.Rule_INFO},
+									{Severity: rpc.Rule_HINT},
+								},
+							},
+						},
+					},
+					{Status: rpc.Guideline_DEPRECATED},
+					{Status: rpc.Guideline_DISABLED},
+				},
+			},
+		},
+		// Test: Multiple rules are defined in multiple guidelines, check conformance report gets generated accurately.
+		{
+			desc: "Multiple guidelines",
+			linterResponse: &rpc.LinterResponse{
+				Lint: &rpc.Lint{
+					Name: "sample-result",
+					Files: []*rpc.LintFile{
+						{
+							FilePath: "test-result-file",
+							Problems: []*rpc.LintProblem{
+								{
+									Message:    "tag-description violated",
+									RuleId:     "tag-description",
+									Suggestion: "fix tag-description",
+								},
+								{
+									Message:    "operation-description violated",
+									RuleId:     "operation-description",
+									Suggestion: "fix operation-description",
+								},
+								{
+									Message:    "info-description violated",
+									RuleId:     "info-description",
+									Suggestion: "fix info-description",
+								},
+								{
+									Message:    "description-contains-no-tags violated",
+									RuleId:     "description-contains-no-tags",
+									Suggestion: "fix description-contains-no-tags",
+								},
+								{
+									Message:    "no-$ref-siblings violated",
+									RuleId:     "no-$ref-siblings",
+									Suggestion: "fix no-$ref-siblings",
+								},
+							},
+						},
+					},
+				},
+			},
+			linterMetadata: &linterMetadata{
+				name:  "sample-linter",
+				rules: []string{"operation-description", "tag-description", "info-description", "no-$ref-siblings"},
+				rulesMetadata: map[string]*ruleMetadata{
+					"operation-description": {
+						guidelineRule: &rpc.Rule{
+							Id:       "operationdescription",
+							Severity: rpc.Rule_ERROR,
+						},
+						guideline: &rpc.Guideline{
+							Id:     "descriptionproperties",
+							Status: rpc.Guideline_ACTIVE,
+						},
+					},
+					"tag-description": {
+						guidelineRule: &rpc.Rule{
+							Id:       "tagdescription",
+							Severity: rpc.Rule_WARNING,
+						},
+						guideline: &rpc.Guideline{
+							Id:     "descriptionproperties",
+							Status: rpc.Guideline_ACTIVE,
+						},
+					},
+					"info-description": {
+						guidelineRule: &rpc.Rule{
+							Id:       "infodescription",
+							Severity: rpc.Rule_WARNING,
+						},
+						guideline: &rpc.Guideline{
+							Id:     "descriptionproperties",
+							Status: rpc.Guideline_ACTIVE,
+						},
+					},
+					"no-$ref-siblings": {
+						guidelineRule: &rpc.Rule{
+							Id:       "norefsiblings",
+							Severity: rpc.Rule_ERROR,
+						},
+						guideline: &rpc.Guideline{
+							Id:     "refproperties",
+							Status: rpc.Guideline_PROPOSED,
+						},
+					},
+				},
+			},
+			wantReport: &rpc.ConformanceReport{
+				Name:           fmt.Sprintf("%s/artifacts/conformance-%s", specName, styleguideId),
+				StyleguideName: styleguideId,
+				GuidelineReportGroups: []*rpc.GuidelineReportGroup{
+					{Status: rpc.Guideline_STATUS_UNSPECIFIED},
+					{
+						Status: rpc.Guideline_PROPOSED,
+						GuidelineReports: []*rpc.GuidelineReport{
+							{
+								GuidelineId: "refproperties",
+								RuleReportGroups: []*rpc.RuleReportGroup{
+									{Severity: rpc.Rule_SEVERITY_UNSPECIFIED},
+									{
+										Severity: rpc.Rule_ERROR,
+										RuleReports: []*rpc.RuleReport{
+											{
+												RuleId:     "norefsiblings",
+												SpecName:   specName,
+												FileName:   "test-result-file",
+												Suggestion: "fix no-$ref-siblings",
+											},
+										},
+									},
+									{Severity: rpc.Rule_WARNING},
+									{Severity: rpc.Rule_INFO},
+									{Severity: rpc.Rule_HINT},
+								},
+							},
+						},
+					},
+					{
+						Status: rpc.Guideline_ACTIVE,
+						GuidelineReports: []*rpc.GuidelineReport{
+							{
+								GuidelineId: "descriptionproperties",
+								RuleReportGroups: []*rpc.RuleReportGroup{
+									{Severity: rpc.Rule_SEVERITY_UNSPECIFIED},
+									{
+										Severity: rpc.Rule_ERROR,
+										RuleReports: []*rpc.RuleReport{
+											{
+												RuleId:     "operationdescription",
+												SpecName:   specName,
+												FileName:   "test-result-file",
+												Suggestion: "fix operation-description",
+											},
+										},
+									},
+									{
+										Severity: rpc.Rule_WARNING,
+										RuleReports: []*rpc.RuleReport{
+											{
+												RuleId:     "tagdescription",
+												SpecName:   specName,
+												FileName:   "test-result-file",
+												Suggestion: "fix tag-description",
+											},
+											{
+												RuleId:     "infodescription",
+												SpecName:   specName,
+												FileName:   "test-result-file",
+												Suggestion: "fix info-description",
+											},
+										},
+									},
+									{Severity: rpc.Rule_INFO},
+									{Severity: rpc.Rule_HINT},
+								},
+							},
+						},
+					},
+					{Status: rpc.Guideline_DEPRECATED},
+					{Status: rpc.Guideline_DISABLED},
+				},
+			},
+		},
+	}
+
+	for _, test := range tests {
+		t.Run(test.desc, func(t *testing.T) {
+			ctx := context.Background()
+
+			task := &ComputeConformanceTask{
+				Spec:         &rpc.ApiSpec{Name: specName},
+				StyleguideId: styleguideId,
+			}
+
+			gotReport := initializeConformanceReport(task.Spec.GetName(), task.StyleguideId)
+			guidelineReportsMap := make(map[string]int)
+			task.computeConformanceReport(ctx, gotReport, guidelineReportsMap, test.linterResponse, test.linterMetadata)
+
+			opts := cmp.Options{
+				protocmp.IgnoreFields(&rpc.RuleReport{}),
+				protocmp.Transform(),
+				cmpopts.SortSlices(func(a, b string) bool { return a < b }),
+			}
+			if !cmp.Equal(test.wantReport, gotReport, opts) {
+				t.Errorf("GetDiff returned unexpected diff (-want +got):\n%s", cmp.Diff(test.wantReport, gotReport, opts))
+			}
+		})
+	}
+}
+
+// Test the scenario where there are pre-existing entries in the conformance report from other linters.
+func TestPreExistingConformanceReport(t *testing.T) {
+	linterResponse := &rpc.LinterResponse{
+		Lint: &rpc.Lint{
+			Name: "sample-result",
+			Files: []*rpc.LintFile{
+				{
+					FilePath: "test-result-file",
+					Problems: []*rpc.LintProblem{
+						{
+							Message:    "operation-description violated",
+							RuleId:     "operation-description",
+							Suggestion: "fix operation-description",
+						},
+					},
+				},
+			},
+		},
+	}
+
+	linterMetadata := &linterMetadata{
+		name:  "sample-linter",
+		rules: []string{"operation-description"},
+		rulesMetadata: map[string]*ruleMetadata{
+			"operation-description": {
+				guidelineRule: &rpc.Rule{
+					Id:       "operationdescription",
+					Severity: rpc.Rule_ERROR,
+				},
+				guideline: &rpc.Guideline{
+					Id:     "descriptionproperties",
+					Status: rpc.Guideline_ACTIVE,
+				},
+			},
+		},
+	}
+
+	preexistingReport := &rpc.ConformanceReport{
+		Name:           fmt.Sprintf("%s/artifacts/conformance-%s", specName, styleguideId),
+		StyleguideName: styleguideId,
+		GuidelineReportGroups: []*rpc.GuidelineReportGroup{
+			{Status: rpc.Guideline_STATUS_UNSPECIFIED},
+			{Status: rpc.Guideline_PROPOSED},
+			{
+				Status: rpc.Guideline_ACTIVE,
+				GuidelineReports: []*rpc.GuidelineReport{
+					{
+						GuidelineId: "descriptionproperties",
+						RuleReportGroups: []*rpc.RuleReportGroup{
+							{Severity: rpc.Rule_SEVERITY_UNSPECIFIED},
+							{
+								Severity: rpc.Rule_ERROR,
+								RuleReports: []*rpc.RuleReport{
+									{
+										RuleId:     "tagdescription",
+										SpecName:   specName,
+										FileName:   "test-result-file",
+										Suggestion: "fix tag-description",
+									},
+								},
+							},
+							{Severity: rpc.Rule_WARNING},
+							{Severity: rpc.Rule_INFO},
+							{Severity: rpc.Rule_HINT},
+						},
+					},
+				},
+			},
+			{Status: rpc.Guideline_DEPRECATED},
+			{Status: rpc.Guideline_DISABLED},
+		},
+	}
+
+	guidelineReportsMap := map[string]int{
+		"descriptionproperties": 0,
+	}
+
+	wantReport := &rpc.ConformanceReport{
+		Name:           fmt.Sprintf("%s/artifacts/conformance-%s", specName, styleguideId),
+		StyleguideName: styleguideId,
+		GuidelineReportGroups: []*rpc.GuidelineReportGroup{
+			{Status: rpc.Guideline_STATUS_UNSPECIFIED},
+			{Status: rpc.Guideline_PROPOSED},
+			{
+				Status: rpc.Guideline_ACTIVE,
+				GuidelineReports: []*rpc.GuidelineReport{
+					{
+						GuidelineId: "descriptionproperties",
+						RuleReportGroups: []*rpc.RuleReportGroup{
+							{Severity: rpc.Rule_SEVERITY_UNSPECIFIED},
+							{
+								Severity: rpc.Rule_ERROR,
+								RuleReports: []*rpc.RuleReport{
+									{
+										RuleId:     "tagdescription",
+										SpecName:   specName,
+										FileName:   "test-result-file",
+										Suggestion: "fix tag-description",
+									},
+									{
+										RuleId:     "operationdescription",
+										SpecName:   specName,
+										FileName:   "test-result-file",
+										Suggestion: "fix operation-description",
+									},
+								},
+							},
+							{Severity: rpc.Rule_WARNING},
+							{Severity: rpc.Rule_INFO},
+							{Severity: rpc.Rule_HINT},
+						},
+					},
+				},
+			},
+			{Status: rpc.Guideline_DEPRECATED},
+			{Status: rpc.Guideline_DISABLED},
+		},
+	}
+
+	ctx := context.Background()
+
+	task := &ComputeConformanceTask{
+		Spec:         &rpc.ApiSpec{Name: specName},
+		StyleguideId: styleguideId,
+	}
+
+	task.computeConformanceReport(ctx, preexistingReport, guidelineReportsMap, linterResponse, linterMetadata)
+
+	opts := cmp.Options{
+		protocmp.IgnoreFields(&rpc.RuleReport{}),
+		protocmp.Transform(),
+		cmpopts.SortSlices(func(a, b string) bool { return a < b }),
+	}
+	if !cmp.Equal(wantReport, preexistingReport, opts) {
+		t.Errorf("GetDiff returned unexpected diff (-want +got):\n%s", cmp.Diff(wantReport, preexistingReport, opts))
+	}
+
+}
diff --git a/cmd/registry/conformance/linter_test.go b/cmd/registry/conformance/linter_test.go
new file mode 100644
index 0000000..27b2160
--- /dev/null
+++ b/cmd/registry/conformance/linter_test.go
@@ -0,0 +1,292 @@
+// Copyright 2022 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package conformance
+
+import (
+	"testing"
+
+	"github.com/apigee/registry/rpc"
+	"github.com/google/go-cmp/cmp"
+	"github.com/google/go-cmp/cmp/cmpopts"
+	"google.golang.org/protobuf/testing/protocmp"
+)
+
+var noRefSiblingsRule = &rpc.Rule{
+	Id:             "norefsiblings",
+	Linter:         "sample",
+	LinterRulename: "no-$ref-siblings",
+	Severity:       rpc.Rule_ERROR,
+}
+
+var noRefCyclesRule = &rpc.Rule{
+	Id:             "norefcycles",
+	Linter:         "spectral",
+	LinterRulename: "no-ref-cycles",
+	Severity:       rpc.Rule_ERROR,
+}
+
+var operaationDescriptionRule = &rpc.Rule{
+	Id:             "operationdescription",
+	Linter:         "spectral",
+	LinterRulename: "operation-description",
+	Severity:       rpc.Rule_ERROR,
+}
+
+func TestGenerateLinterMetadata(t *testing.T) {
+	tests := []struct {
+		desc       string
+		styleguide *rpc.StyleGuide
+		want       map[string]*linterMetadata
+		wantErr    bool
+	}{
+		{
+			desc: "Normal case",
+			styleguide: &rpc.StyleGuide{
+				Id:        "openapitest",
+				MimeTypes: []string{"application/x.openapi+gzip;version=3.0.0"},
+				Guidelines: []*rpc.Guideline{
+					{
+						Id:     "refproperties",
+						Rules:  []*rpc.Rule{noRefSiblingsRule},
+						Status: rpc.Guideline_ACTIVE,
+					},
+				},
+			},
+			want: map[string]*linterMetadata{
+				"sample": {
+					name:  "sample",
+					rules: []string{noRefSiblingsRule.GetLinterRulename()},
+					rulesMetadata: map[string]*ruleMetadata{
+						noRefSiblingsRule.GetLinterRulename(): {
+							guidelineRule: noRefSiblingsRule,
+							guideline: &rpc.Guideline{
+								Id:     "refproperties",
+								Rules:  []*rpc.Rule{noRefSiblingsRule},
+								Status: rpc.Guideline_ACTIVE,
+							},
+						},
+					},
+				},
+			},
+		},
+		{
+			desc: "Multiple linters",
+			styleguide: &rpc.StyleGuide{
+				Id:        "openapitest",
+				MimeTypes: []string{"application/x.openapi+gzip;version=3.0.0"},
+				Guidelines: []*rpc.Guideline{
+					{
+						Id:     "descriptionproperties",
+						Rules:  []*rpc.Rule{noRefSiblingsRule, noRefCyclesRule},
+						Status: rpc.Guideline_ACTIVE,
+					},
+				},
+			},
+			want: map[string]*linterMetadata{
+				"sample": {
+					name:  "sample",
+					rules: []string{noRefSiblingsRule.GetLinterRulename()},
+					rulesMetadata: map[string]*ruleMetadata{
+						noRefSiblingsRule.GetLinterRulename(): {
+							guidelineRule: noRefSiblingsRule,
+							guideline: &rpc.Guideline{
+								Id:     "descriptionproperties",
+								Rules:  []*rpc.Rule{noRefSiblingsRule, noRefCyclesRule},
+								Status: rpc.Guideline_ACTIVE,
+							},
+						},
+					},
+				},
+				"spectral": {
+					name:  "spectral",
+					rules: []string{noRefCyclesRule.GetLinterRulename()},
+					rulesMetadata: map[string]*ruleMetadata{
+						noRefCyclesRule.GetLinterRulename(): {
+							guidelineRule: noRefCyclesRule,
+							guideline: &rpc.Guideline{
+								Id:     "descriptionproperties",
+								Rules:  []*rpc.Rule{noRefSiblingsRule, noRefCyclesRule},
+								Status: rpc.Guideline_ACTIVE,
+							},
+						},
+					},
+				},
+			},
+		},
+		{
+			desc: "Multiple linters guidelines",
+			styleguide: &rpc.StyleGuide{
+				Id:        "openapitest",
+				MimeTypes: []string{"application/x.openapi+gzip;version=3.0.0"},
+				Guidelines: []*rpc.Guideline{
+					{
+						Id:     "refproperties",
+						Rules:  []*rpc.Rule{noRefSiblingsRule, noRefCyclesRule},
+						Status: rpc.Guideline_ACTIVE,
+					},
+					{
+						Id:     "descriptionproperties",
+						Rules:  []*rpc.Rule{operaationDescriptionRule},
+						Status: rpc.Guideline_PROPOSED,
+					},
+				},
+			},
+			want: map[string]*linterMetadata{
+				"sample": {
+					name:  "sample",
+					rules: []string{noRefSiblingsRule.GetLinterRulename()},
+					rulesMetadata: map[string]*ruleMetadata{
+						noRefSiblingsRule.GetLinterRulename(): {
+							guidelineRule: noRefSiblingsRule,
+							guideline: &rpc.Guideline{
+								Id:     "refproperties",
+								Rules:  []*rpc.Rule{noRefSiblingsRule, noRefCyclesRule},
+								Status: rpc.Guideline_ACTIVE,
+							},
+						},
+					},
+				},
+				"spectral": {
+					name:  "spectral",
+					rules: []string{noRefCyclesRule.GetLinterRulename(), operaationDescriptionRule.GetLinterRulename()},
+					rulesMetadata: map[string]*ruleMetadata{
+						noRefCyclesRule.GetLinterRulename(): {
+							guidelineRule: noRefCyclesRule,
+							guideline: &rpc.Guideline{
+								Id:     "refproperties",
+								Rules:  []*rpc.Rule{noRefSiblingsRule, noRefCyclesRule},
+								Status: rpc.Guideline_ACTIVE,
+							},
+						},
+						operaationDescriptionRule.GetLinterRulename(): {
+							guidelineRule: operaationDescriptionRule,
+							guideline: &rpc.Guideline{
+								Id:     "descriptionproperties",
+								Rules:  []*rpc.Rule{operaationDescriptionRule},
+								Status: rpc.Guideline_PROPOSED,
+							},
+						},
+					},
+				},
+			},
+		},
+		{
+			desc: "Empty metadata",
+			styleguide: &rpc.StyleGuide{
+				Id:        "openapitest",
+				MimeTypes: []string{"application/x.openapi+gzip;version=3.0.0"},
+				Guidelines: []*rpc.Guideline{
+					{
+						Id:     "refproperties",
+						Rules:  []*rpc.Rule{},
+						Status: rpc.Guideline_ACTIVE,
+					},
+				},
+			},
+			want:    nil,
+			wantErr: true,
+		},
+		{
+			desc: "Missing linterName",
+			styleguide: &rpc.StyleGuide{
+				Id:        "openapitest",
+				MimeTypes: []string{"application/x.openapi+gzip;version=3.0.0"},
+				Guidelines: []*rpc.Guideline{
+					{
+						Id: "refproperties",
+						Rules: []*rpc.Rule{
+							{
+								Id:             "norefsiblings",
+								LinterRulename: "no-$ref-siblings",
+								Severity:       rpc.Rule_ERROR,
+							},
+						},
+						Status: rpc.Guideline_ACTIVE,
+					},
+				},
+			},
+			want:    nil,
+			wantErr: true,
+		},
+		{
+			desc: "Missing linterRuleName",
+			styleguide: &rpc.StyleGuide{
+				Id:        "openapitest",
+				MimeTypes: []string{"application/x.openapi+gzip;version=3.0.0"},
+				Guidelines: []*rpc.Guideline{
+					{
+						Id: "refproperties",
+						Rules: []*rpc.Rule{
+							{
+								Id:       "norefsiblings",
+								Linter:   "spectral",
+								Severity: rpc.Rule_ERROR,
+							},
+							noRefCyclesRule,
+						},
+						Status: rpc.Guideline_ACTIVE,
+					},
+				},
+			},
+			want: map[string]*linterMetadata{
+				"spectral": {
+					name:  "spectral",
+					rules: []string{noRefCyclesRule.GetLinterRulename()},
+					rulesMetadata: map[string]*ruleMetadata{
+						noRefCyclesRule.GetLinterRulename(): {
+							guidelineRule: noRefCyclesRule,
+							guideline: &rpc.Guideline{
+								Id: "refproperties",
+								Rules: []*rpc.Rule{
+									{
+										Id:       "norefsiblings",
+										Linter:   "spectral",
+										Severity: rpc.Rule_ERROR,
+									},
+									noRefCyclesRule,
+								},
+								Status: rpc.Guideline_ACTIVE,
+							},
+						},
+					},
+				},
+			},
+		},
+	}
+
+	for _, test := range tests {
+		t.Run(test.desc, func(t *testing.T) {
+
+			got, err := GenerateLinterMetadata(test.styleguide)
+
+			if test.wantErr && err == nil {
+				t.Fatalf("Expected GenerateLinterMetadata() to return an error")
+			} else if !test.wantErr && err != nil {
+				t.Fatalf("Unexpected error from GenerateLinterMetadata(): %s", err)
+			}
+
+			opts := cmp.Options{
+				protocmp.Transform(),
+				cmpopts.SortSlices(func(a, b string) bool { return a < b }),
+				cmp.AllowUnexported(linterMetadata{}),
+				cmp.AllowUnexported(ruleMetadata{}),
+				cmpopts.SortMaps(func(a, b string) bool { return a < b }),
+			}
+			if !cmp.Equal(test.want, got, opts) {
+				t.Errorf("GetDiff returned unexpected diff (-want +got):\n%s", cmp.Diff(test.want, got, opts))
+			}
+		})
+	}
+}