Repository: stoplightio/spectral
Pull request #798 (2019-11-21 00:59:08)

feat: report unrecognized formats
Closes https://github.com/stoplightio/spectral/issues/678

**Checklist**

- [x] Tests added / updated
- [x] Docs added / updated

**Does this PR introduce a breaking change?**

- [ ] Yes
- [x] No



philsturgeon: Slight concern here. If I am working on a Spectral ruleset in studio it will not have a format, so im going to see complaints about a lack of format all the time I work on it.

Any other .json file I work on in studio will also complain about a lack of format. 

Is this a good feature? 
P0lip: > Slight concern here. If I am working on a Spectral ruleset in studio it will not have a format, so im going to see complaints about a lack of format all the time I work on it.

Should not be a problem, since we don't use Spectral for linting Spectral ruleset.

> Any other .json file I work on in studio will also complain about a lack of format.

True, but similarly to above, I can disable linting of such documents.

Either way, I hear your concerns.
Perhaps we make it opt-in (flag?) and exclusive to CLI?
marbemac: Suggest add a method to public API along the lines `supportsFormat(id: string): boolean` - Studio (or other implementations) could use this to check before running spectral on a file. Thoughts?
nulltoken: @P0lip Wow. Thanks for making that happening!
philsturgeon: @P0lip yeah this could either be a CLI default, or opt-in for the CLI, or opt-in everywhere, but I don't think it should be default eerywhere.

I mentioned rulesets because in Studio we do seem to lint rulesets, and I asked about random json files because I noticed it linting random json files 😓 

Having studio only applying rules it knows to formats it recognizes is a good feature, but I can understand the CLI users wanting an error or warning if nothing is applied, that's a rather different use case.
P0lip: @philsturgeon @marbemac 
so, what's the final call? :)
opt-in flag for CLI? and an extra method for JS usage?
nulltoken: I'd say that opt-in for cli would actually defeat the purpose of the feature (which was to warn the user that something went sideways).

cf #678 _"Indeed, I may think that my file is perfect and that it passed all the many built-in checks Spectral comes up with. Whereas the sad reality is that it went through NO rule at all."_
philsturgeon: Agreed @nulltoken 

CLI can be default, but what about JS? Default on with a "ignoreUnknownFormat" option?
P0lip: > CLI can be default, but what about JS? Default on with a "ignoreUnknownFormat" option?

Sounds good to me. :+1:
I'd also make it enabled by default.

Are we all set? I wanna wrap this one so badly before 5.0 release :wink: 
philsturgeon: ![image](https://user-images.githubusercontent.com/67381/70352313-29cba280-186b-11ea-9a6a-32faf6cd267c.png)

P0lip: @philsturgeon @marbemac it's ready for another pass
cc @nulltoken 

Patch:
diff --git a/docs/guides/cli.md b/docs/guides/cli.md
index 70f876b8..244e7b6c 100644
--- a/docs/guides/cli.md
+++ b/docs/guides/cli.md
@@ -33,6 +33,7 @@ Other options include:
                                 [string] [choices: "error", "warn", "info", "hint"] [default: "warn"]
   --display-only-failures, -D  only output results equal to or greater than --fail-severity
                                                                            [boolean] [default: false]
+  --ignore-unknown-format      do not warn about unmatched formats         [boolean] [default: false]
   --verbose, -v                increase verbosity                                           [boolean]
   --quiet, -q                  no logging - output only                                     [boolean]
 ```
diff --git a/src/cli/commands/lint.ts b/src/cli/commands/lint.ts
index a44107f6..303f0df1 100644
--- a/src/cli/commands/lint.ts
+++ b/src/cli/commands/lint.ts
@@ -100,6 +100,11 @@ const lintCommand: CommandModule = {
           type: 'boolean',
           default: false,
         },
+        'ignore-unknown-format': {
+          description: 'do not warn about unmatched formats',
+          type: 'boolean',
+          default: false,
+        },
         verbose: {
           alias: 'v',
           description: 'increase verbosity',
@@ -121,6 +126,7 @@ const lintCommand: CommandModule = {
       format,
       output,
       encoding,
+      ignoreUnknownFormat,
       ...config
     } = (args as unknown) as ILintConfig & {
       documents: Array<number | string>;
@@ -132,6 +138,7 @@ const lintCommand: CommandModule = {
       format,
       output,
       encoding,
+      ignoreUnknownFormat,
       ruleset,
       ...pick<Partial<ILintConfig>, keyof ILintConfig>(config, ['skipRule', 'verbose', 'quiet', 'resolver']),
     })
diff --git a/src/cli/services/linter/linter.ts b/src/cli/services/linter/linter.ts
index ea9759b1..4b3f4e1e 100644
--- a/src/cli/services/linter/linter.ts
+++ b/src/cli/services/linter/linter.ts
@@ -84,6 +84,7 @@ export async function lint(documents: Array<number | string>, flags: ILintConfig
 
     results.push(
       ...(await spectral.run(parsedResult, {
+        ignoreUnknownFormat: flags.ignoreUnknownFormat,
         resolve: {
           documentUri: typeof targetUri === 'number' ? void 0 : targetUri,
         },
diff --git a/src/spectral.ts b/src/spectral.ts
index bab55ead..fceb1f21 100644
--- a/src/spectral.ts
+++ b/src/spectral.ts
@@ -2,7 +2,7 @@ import { getLocationForJsonPath as getLocationForJsonPathJson, JsonParserResult,
 import { Resolver } from '@stoplight/json-ref-resolver';
 import { ICache, IUriParser } from '@stoplight/json-ref-resolver/types';
 import { extname, normalize } from '@stoplight/path';
-import { Dictionary, IDiagnostic, Optional } from '@stoplight/types';
+import { DiagnosticSeverity, Dictionary, IDiagnostic, Optional } from '@stoplight/types';
 import { getLocationForJsonPath as getLocationForJsonPathYaml, YamlParserResult } from '@stoplight/yaml';
 import { merge } from 'lodash';
 
@@ -96,21 +96,24 @@ export class Spectral {
       this._parsedRefs,
     );
 
+    const validationResults = [...refDiagnostics, ...results, ...formatResolverErrors(resolved)];
+
     if (resolved.formats === void 0) {
-      const foundFormats = Object.keys(this.formats).filter(format => this.formats[format](resolved.resolved));
-      resolved.formats = foundFormats.length === 0 ? null : foundFormats;
+      const registeredFormats = Object.keys(this.formats);
+      const foundFormats = registeredFormats.filter(format => this.formats[format](resolved.resolved));
+      if (foundFormats.length === 0 && opts.ignoreUnknownFormat !== true) {
+        resolved.formats = null;
+        if (registeredFormats.length > 0) {
+          validationResults.push(this._generateUnrecognizedFormatError(parsedResult));
+        }
+      } else {
+        resolved.formats = foundFormats;
+      }
     }
 
-    const validationResults = [
-      ...refDiagnostics,
-      ...results,
-      ...formatResolverErrors(resolved),
-      ...runRules(resolved, this.rules, this.functions),
-    ];
-
     return {
       resolved: resolved.resolved,
-      results: validationResults,
+      results: [...validationResults, ...runRules(resolved, this.rules, this.functions)],
     };
   }
 
@@ -215,6 +218,23 @@ export class Spectral {
 
     return resolveOpts;
   };
+
+  private _generateUnrecognizedFormatError(parsedResult: IParsedResult): IRuleResult {
+    return {
+      range: parsedResult.getLocationForJsonPath(parsedResult.parsed, [], true)?.range || {
+        start: { character: 0, line: 0 },
+        end: { character: 0, line: 0 },
+      },
+
+      message: `The provided document does not match any of the registered formats [${Object.keys(this.formats).join(
+        ', ',
+      )}]`,
+      code: 'unrecognized-format',
+      severity: DiagnosticSeverity.Warning,
+      source: parsedResult.source,
+      path: [],
+    };
+  }
 }
 
 export const isParsedResult = (obj: any): obj is IParsedResult => {
diff --git a/src/types/config.ts b/src/types/config.ts
index 8f723fc8..d584c7ec 100644
--- a/src/types/config.ts
+++ b/src/types/config.ts
@@ -18,6 +18,7 @@ export interface ILintConfig {
   resolver?: string;
   ruleset?: string[];
   skipRule?: string[];
+  ignoreUnknownFormat: boolean;
   verbose?: boolean;
   quiet?: boolean;
 }
diff --git a/src/types/spectral.ts b/src/types/spectral.ts
index 323526ce..87663874 100644
--- a/src/types/spectral.ts
+++ b/src/types/spectral.ts
@@ -34,6 +34,7 @@ export interface IConstructorOpts {
 }
 
 export interface IRunOpts {
+  ignoreUnknownFormat?: boolean;
   resolve?: {
     documentUri?: string;
   };


Test patch:
diff --git a/src/__tests__/linter.jest.test.ts b/src/__tests__/linter.jest.test.ts
index cbc74449..06f9f8f5 100644
--- a/src/__tests__/linter.jest.test.ts
+++ b/src/__tests__/linter.jest.test.ts
@@ -1,7 +1,6 @@
 import * as path from '@stoplight/path';
 import { join } from '@stoplight/path';
 import { DiagnosticSeverity } from '@stoplight/types';
-import { isOpenApiv2, isOpenApiv3 } from '../formats';
 import { httpAndFileResolver } from '../resolvers/http-and-file';
 import { Spectral } from '../spectral';
 
@@ -13,8 +12,6 @@ describe('Linter', () => {
 
   beforeEach(() => {
     spectral = new Spectral();
-    spectral.registerFormat('oas3', isOpenApiv3);
-    spectral.registerFormat('oas2', isOpenApiv2);
   });
 
   it('should make use of custom functions', async () => {
diff --git a/src/__tests__/linter.test.ts b/src/__tests__/linter.test.ts
index 6233e99b..24d545b1 100644
--- a/src/__tests__/linter.test.ts
+++ b/src/__tests__/linter.test.ts
@@ -41,20 +41,18 @@ describe('linter', () => {
 
   beforeEach(() => {
     spectral = new Spectral();
-    spectral.registerFormat('oas3', isOpenApiv3);
     spectral.registerFormat('oas2', isOpenApiv2);
+    spectral.registerFormat('oas3', isOpenApiv3);
   });
 
-  test('should not lint if passed in value is not an object', async () => {
+  test('should not throw if passed in value is not an object', () => {
     const fakeLintingFunction = jest.fn();
     spectral.setFunctions({
       [fnName]: fakeLintingFunction,
     });
     spectral.setRules(rules);
 
-    const result = await spectral.run('123');
-
-    expect(result).toHaveLength(0);
+    return expect(spectral.run('123')).resolves.toBeTruthy();
   });
 
   test('should return all properties matching 4xx response code', async () => {
@@ -84,16 +82,19 @@ describe('linter', () => {
       },
     });
 
-    const result = await spectral.run({
-      responses: {
-        '200': {
-          name: 'ok',
-        },
-        '404': {
-          name: 'not found',
+    const result = await spectral.run(
+      {
+        responses: {
+          '200': {
+            name: 'ok',
+          },
+          '404': {
+            name: 'not found',
+          },
         },
       },
-    });
+      { ignoreUnknownFormat: true },
+    );
 
     expect(result).toEqual([
       {
@@ -136,9 +137,12 @@ describe('linter', () => {
       },
     });
 
-    const result = await spectral.run({
-      x: true,
-    });
+    const result = await spectral.run(
+      {
+        x: true,
+      },
+      { ignoreUnknownFormat: true },
+    );
 
     expect(result[0]).toHaveProperty('severity', DiagnosticSeverity.Hint);
   });
@@ -215,10 +219,13 @@ describe('linter', () => {
       },
     });
 
-    const result = await spectral.run({
-      x: false,
-      y: '',
-    });
+    const result = await spectral.run(
+      {
+        x: false,
+        y: '',
+      },
+      { ignoreUnknownFormat: true },
+    );
 
     expect(result).toEqual([
       expect.objectContaining({
@@ -338,6 +345,10 @@ describe('linter', () => {
     });
 
     expect(result).toEqual([
+      expect.objectContaining({
+        code: 'unrecognized-format',
+        message: 'The provided document does not match any of the registered formats [oas2, oas3]',
+      }),
       expect.objectContaining({
         code: 'rule3',
       }),
@@ -452,12 +463,68 @@ describe('linter', () => {
     });
 
     expect(result).toEqual([
+      {
+        message: 'The provided document does not match any of the registered formats [oas2, oas3, foo-bar]',
+        path: [],
+        range: expect.any(Object),
+        severity: DiagnosticSeverity.Warning,
+        code: 'unrecognized-format',
+      },
       expect.objectContaining({
         code: 'rule3',
       }),
     ]);
   });
 
+  test('given a string input, should warn about unmatched formats', async () => {
+    const result = await spectral.run('test');
+
+    expect(result).toEqual([
+      {
+        code: 'unrecognized-format',
+        message: 'The provided document does not match any of the registered formats [oas2, oas3]',
+        path: [],
+        range: {
+          end: {
+            character: 4,
+            line: 0,
+          },
+          start: {
+            character: 0,
+            line: 0,
+          },
+        },
+        severity: DiagnosticSeverity.Warning,
+      },
+    ]);
+  });
+
+  test('given ignoreUnknownFormat, should not warn about unmatched formats', async () => {
+    spectral.registerFormat('foo-bar', obj => typeof obj === 'object' && obj !== null && 'foo-bar' in obj);
+
+    spectral.setRules({
+      rule1: {
+        given: '$.x',
+        formats: ['foo-bar'],
+        severity: 'error',
+        then: {
+          function: 'truthy',
+        },
+      },
+    });
+
+    const result = await spectral.run(
+      {
+        'bar-foo': true,
+        x: true,
+        y: '',
+      },
+      { ignoreUnknownFormat: true },
+    );
+
+    expect(result).toEqual([]);
+  });
+
   test('should include parser diagnostics', async () => {
     await spectral.loadRuleset('spectral:oas');
 
@@ -512,6 +579,8 @@ responses:: !!foo
   });
 
   test('should report a valid line number for json paths containing escaped slashes', async () => {
+    spectral.registerFormat('oas2', isOpenApiv2);
+    spectral.registerFormat('oas3', isOpenApiv3);
     await spectral.loadRuleset('spectral:oas');
 
     const result = await spectral.run(studioFixture);
@@ -537,6 +606,8 @@ responses:: !!foo
   });
 
   test('should remove all redundant ajv errors', async () => {
+    spectral.registerFormat('oas2', isOpenApiv2);
+    spectral.registerFormat('oas3', isOpenApiv3);
     await spectral.loadRuleset('spectral:oas');
 
     const result = await spectral.run(invalidSchema);
@@ -578,6 +649,8 @@ responses:: !!foo
   });
 
   test('should report invalid schema $refs', async () => {
+    spectral.registerFormat('oas2', isOpenApiv2);
+    spectral.registerFormat('oas3', isOpenApiv3);
     await spectral.loadRuleset('spectral:oas');
 
     const result = await spectral.run(todosInvalid);
@@ -632,10 +705,13 @@ responses:: !!foo
 
   describe('reports duplicated properties for', () => {
     test('JSON format', async () => {
-      const result = await spectral.run({
-        parsed: parseWithPointers('{"foo":true,"foo":false}', { ignoreDuplicateKeys: false }),
-        getLocationForJsonPath,
-      });
+      const result = await spectral.run(
+        {
+          parsed: parseWithPointers('{"foo":true,"foo":false}', { ignoreDuplicateKeys: false }),
+          getLocationForJsonPath,
+        },
+        { ignoreUnknownFormat: true },
+      );
 
       expect(result).toEqual([
         {
@@ -658,7 +734,7 @@ responses:: !!foo
     });
 
     test('YAML format', async () => {
-      const result = await spectral.run(`foo: bar\nfoo: baz`);
+      const result = await spectral.run(`foo: bar\nfoo: baz`, { ignoreUnknownFormat: true });
 
       expect(result).toEqual([
         {
@@ -682,11 +758,14 @@ responses:: !!foo
   });
 
   test('should report invalid YAML mapping keys', async () => {
-    const results = await spectral.run(`responses:
+    const results = await spectral.run(
+      `responses:
   200:
     description: ''
   '400':
-    description: ''`);
+    description: ''`,
+      { ignoreUnknownFormat: true },
+    );
 
     expect(results).toEqual([
       {
@@ -941,6 +1020,8 @@ responses:: !!foo
   });
 
   test('should evaluate {{path}} in validation messages', async () => {
+    spectral.registerFormat('oas2', isOpenApiv2);
+    spectral.registerFormat('oas3', isOpenApiv3);
     await spectral.loadRuleset('spectral:oas');
     spectral.setRules({
       'oas3-schema': {
@@ -961,6 +1042,8 @@ responses:: !!foo
   });
 
   test('should report ref siblings', async () => {
+    spectral.registerFormat('oas2', isOpenApiv2);
+    spectral.registerFormat('oas3', isOpenApiv3);
     await spectral.loadRuleset('spectral:oas');
 
     const results = await spectral.run({
diff --git a/src/cli/commands/__tests__/lint.test.ts b/src/cli/commands/__tests__/lint.test.ts
index 748283aa..15cf7d4f 100644
--- a/src/cli/commands/__tests__/lint.test.ts
+++ b/src/cli/commands/__tests__/lint.test.ts
@@ -76,6 +76,7 @@ describe('lint', () => {
       expect(lint).toBeCalledWith([0], {
         encoding: 'utf8',
         format: 'stylish',
+        ignoreUnknownFormat: false,
       });
     });
   });
@@ -91,6 +92,7 @@ describe('lint', () => {
     expect(lint).toBeCalledWith([doc], {
       encoding: 'utf8',
       format: 'stylish',
+      ignoreUnknownFormat: false,
     });
   });
 
@@ -100,6 +102,7 @@ describe('lint', () => {
     expect(lint).toBeCalledWith([doc], {
       encoding: 'utf16',
       format: 'stylish',
+      ignoreUnknownFormat: false,
     });
   });
 
@@ -109,6 +112,7 @@ describe('lint', () => {
     expect(lint).toBeCalledWith([doc], {
       encoding: 'utf16',
       format: 'json',
+      ignoreUnknownFormat: false,
     });
   });
 
@@ -165,6 +169,16 @@ describe('lint', () => {
       skipRule: ['foo', 'bar'],
       encoding: 'utf8',
       format: 'stylish',
+      ignoreUnknownFormat: false,
+    });
+  });
+
+  it('passes ignore-unknown-format to lint', async () => {
+    await run('lint --ignore-unknown-format ./__fixtures__/empty-oas2-document.json');
+    expect(lint).toHaveBeenCalledWith([expect.any(String)], {
+      encoding: 'utf8',
+      format: 'stylish',
+      ignoreUnknownFormat: true,
     });
   });
 
diff --git a/test-harness/scenarios/custom-ref-resolver.scenario b/test-harness/scenarios/custom-ref-resolver.scenario
index 6a673a0f..5b439320 100644
--- a/test-harness/scenarios/custom-ref-resolver.scenario
+++ b/test-harness/scenarios/custom-ref-resolver.scenario
@@ -10,6 +10,6 @@ rules:
       field: user
       function: truthy
 ====command====
-{bin} lint {document} --ruleset {asset:ruleset} --resolver ./resolvers/custom.js
+{bin} lint {document} --ruleset {asset:ruleset} --resolver ./resolvers/custom.js --ignore-unknown-format
 ====stdout====
 No results with a severity of 'warn' or higher found!
diff --git a/test-harness/scenarios/help-no-document.scenario b/test-harness/scenarios/help-no-document.scenario
index d0572fe6..6d6ac335 100644
--- a/test-harness/scenarios/help-no-document.scenario
+++ b/test-harness/scenarios/help-no-document.scenario
@@ -21,5 +21,6 @@ Options:
   --skip-rule, -s              ignore certain rules if they are causing trouble  [string]
   --fail-severity, -F          results of this level or above will trigger a failure exit code  [string] [choices: "error", "warn", "info", "hint"] [default: "warn"]
   --display-only-failures, -D  only output results equal to or greater than --fail-severity  [boolean] [default: false]
+  --ignore-unknown-format      do not warn about unmatched formats  [boolean] [default: false]
   --verbose, -v                increase verbosity  [boolean]
   --quiet, -q                  no logging - output only  [boolean]
diff --git a/test-harness/scenarios/ignored-unrecognized-format.scenario b/test-harness/scenarios/ignored-unrecognized-format.scenario
new file mode 100644
index 00000000..e0f4bd6c
--- /dev/null
+++ b/test-harness/scenarios/ignored-unrecognized-format.scenario
@@ -0,0 +1,9 @@
+====test====
+Does not report unrecognized formats given --ignore-unknown-format
+====document====
+openapi: 4.0.0
+info: {}
+====command====
+{bin} lint {document} --ignore-unknown-format
+====stdout====
+No results with a severity of 'warn' or higher found!
diff --git a/test-harness/scenarios/proxy-agent.scenario b/test-harness/scenarios/proxy-agent.scenario
index a757a048..5735ecff 100644
--- a/test-harness/scenarios/proxy-agent.scenario
+++ b/test-harness/scenarios/proxy-agent.scenario
@@ -6,9 +6,8 @@ foo:
 ====env====
 PROXY=http://localhost:3001
 ====command====
-{bin} lint {document}
+{bin} lint {document} --ignore-unknown-format
 ====stdout====
-
 {document}
  2:9  error  invalid-ref  FetchError: request to http://localhost:3002/foo.json failed, reason: connect ECONNREFUSED 127.0.0.1:3001
 
diff --git a/test-harness/scenarios/rules-matching-multiple-places.scenario b/test-harness/scenarios/rules-matching-multiple-places.scenario
index 11d77f8e..9c7edd7d 100644
--- a/test-harness/scenarios/rules-matching-multiple-places.scenario
+++ b/test-harness/scenarios/rules-matching-multiple-places.scenario
@@ -33,8 +33,9 @@ rules:
 1
 ====stdout====
 {document}
-  6:15  error  valid-user-properties     must match the pattern '/^string$/'
- 10:15  error  valid-user-properties     must match the pattern '/^string$/'
- 11:11  error  require-user-and-address  schemas.address is not truthy
+  1:1   warning  unrecognized-format       The provided document does not match any of the registered formats [oas2, oas3, json-schema, json-schema-loose, json-schema-draft4, json-schema-draft6, json-schema-draft7, json-schema-2019-09]
+  6:15    error  valid-user-properties     must match the pattern '/^string$/'
+ 10:15    error  valid-user-properties     must match the pattern '/^string$/'
+ 11:11    error  require-user-and-address  schemas.address is not truthy
 
-✖ 3 problems (3 errors, 0 warnings, 0 infos, 0 hints)
+✖ 4 problems (3 errors, 1 warning, 0 infos, 0 hints)
diff --git a/test-harness/scenarios/unrecognized-format.scenario b/test-harness/scenarios/unrecognized-format.scenario
new file mode 100644
index 00000000..9c97b0ff
--- /dev/null
+++ b/test-harness/scenarios/unrecognized-format.scenario
@@ -0,0 +1,12 @@
+====test====
+Reports unrecognized formats
+====document====
+openapi: 4.0.0
+info: {}
+====command====
+{bin} lint {document}
+====stdout====
+{document}
+ 1:1  warning  unrecognized-format  The provided document does not match any of the registered formats [oas2, oas3, json-schema, json-schema-loose, json-schema-draft4, json-schema-draft6, json-schema-draft7, json-schema-2019-09]
+
+✖ 1 problem (0 errors, 1 warning, 0 infos, 0 hints)