Repository: Redocly/redocly-cli
Pull request #842 (2022-08-31 11:58:32)

feat: ability to add other files and folders to push
## What/Why/How?

Fix: #699
Added ability to upload other files and folders. User can provide section `files` in `.redocly.yaml` or as argument `--files`.

## Reference

## Testing

## Screenshots (optional)

<img width="1290" alt="image" src="https://user-images.githubusercontent.com/4946236/187673090-f58a96cd-701c-45a5-bacd-370149eef3f3.png">


## Check yourself

- [x] Code is linted
- [ ] Tested with redoc/reference-docs/workflows
- [x] All new/updated code is covered with tests

## Security

- [x] Security impact of change has been considered
- [x] Code follows company security practices and guidelines


github-actions[bot]: <!-- jest coverage report action for options with hash 3339ae53a38c44382c0fb4491bdb282d -->

## Coverage report 

❌ An unexpected error occurred. For more details, [check console](https://github.com/Redocly/redocly-cli/actions/runs/2963652554) 
```
Error: The process '/usr/local/bin/npm' failed with exit code 1
```

| <div title="Status of coverage:&#10; 🟢 - ok&#10; 🟡 - slightly more than threshold&#10; 🔴 - under the threshold">St.<sup>:grey_question:</sup></div> | Category   | Percentage | Covered / Total |
| :----------------------------------------------------------------------------------------------------------------------------------------------------: | :--------- | :--------- | :-------------: |
|                                                                           🟡                                                                           | Statements | 72.15%     |    3117/4320    |
|                                                                           🟡                                                                           | Branches   | 64.09%     |    1683/2626    |
|                                                                           🟡                                                                           | Functions  | 64.03%     |     509/795     |
|                                                                           🟡                                                                           | Lines      | 72.14%     |    2898/4017    |


<details><summary>Show files with reduced coverage 🔻</summary>
<br/>

| <div title="Status of coverage:&#10; 🟢 - ok&#10; 🟡 - slightly more than threshold&#10; 🔴 - under the threshold">St.<sup>:grey_question:</sup></div> | File                                                           | Statements                                                    | Branches                                                      | Functions                                                      | Lines                                                          |
| :----------------------------------------------------------------------------------------------------------------------------------------------------: | :------------------------------------------------------------- | :------------------------------------------------------------ | :------------------------------------------------------------ | :------------------------------------------------------------- | :------------------------------------------------------------- |
|                                                                           🟡                                                                           | <div title="core/src/config/config.ts">`...` / config.ts</div> | 70.39%                                                        | 76.86%                                                        | <div title="Base coverage is: 65.38%">62.96% (-2.42% 🔻)</div> | 69.78%                                                         |
|                                                                           🟡                                                                           | <div title="cli/src/commands/push.ts">`...` / push.ts</div>    | <div title="Base coverage is: 70.8%">67.12% (-3.68% 🔻)</div> | <div title="Base coverage is: 61.7%">59.18% (-2.52% 🔻)</div> | <div title="Base coverage is: 60%">57.14% (-2.86% 🔻)</div>    | <div title="Base coverage is: 72.87%">69.34% (-3.53% 🔻)</div> |

</details>

## Test suite run failed

<details><summary>Failed tests: 2/482. Failed suites: 1/83.</summary>
<br/>

```
  ● getMergedConfig › should get styleguide defined in "apis" section

    expect(received).toMatchInlineSnapshot(snapshot)

    Snapshot name: `getMergedConfig should get styleguide defined in "apis" section 1`

    - Snapshot  - 0
    + Received  + 1

    @@ -10,10 +10,11 @@
          },
        },
        "configFile": "redocly.yaml",
        "features.mockServer": Object {},
        "features.openapi": Object {},
    +   "files": undefined,
        "organization": "redocly-test",
        "rawConfig": Object {
          "apis": Object {
            "test@v1": Object {
              "root": "resources/pets.yaml",

      53 | describe('getMergedConfig', () => {
      54 |   it('should get styleguide defined in "apis" section', () => {
    > 55 |     expect(getMergedConfig(testConfig, 'test@v1')).toMatchInlineSnapshot(`
         |                                                    ^
      56 |       Config {
      57 |         "apis": Object {
      58 |           "test@v1": Object {

      at Object.<anonymous> (packages/core/src/config/__tests__/config.test.ts:55:52)

  ● getMergedConfig › should handle wrong alias - return the same styleguide, empty features

    expect(received).toMatchInlineSnapshot(snapshot)

    Snapshot name: `getMergedConfig should handle wrong alias - return the same styleguide, empty features 1`

    - Snapshot  - 0
    + Received  + 1

    @@ -10,10 +10,11 @@
          },
        },
        "configFile": "redocly.yaml",
        "features.mockServer": Object {},
        "features.openapi": Object {},
    +   "files": undefined,
        "organization": "redocly-test",
        "rawConfig": Object {
          "apis": Object {
            "test@v1": Object {
              "root": "resources/pets.yaml",

      149 |   });
      150 |   it('should handle wrong alias - return the same styleguide, empty features', () => {
    > 151 |     expect(getMergedConfig(testConfig, 'wrong-alias')).toMatchInlineSnapshot(`
          |                                                        ^
      152 |       Config {
      153 |         "apis": Object {
      154 |           "test@v1": Object {

      at Object.<anonymous> (packages/core/src/config/__tests__/config.test.ts:151:56)

```

</details>


<p align="right">Report generated by <a href="https://github.com/ArtiomTr/jest-coverage-report-action">🧪jest coverage report action</a> from 4042419e4750130a5268e31ce1117465ae106666</p>

tatomyr: Should we also include `files` from other configs when extending? If so, we should add code here: https://github.com/Redocly/redocly-cli/blob/master/packages/core/src/config/utils.ts#L160
iosypov: @adamaltman  @RomanHotsiy  need your reviews, please.

Patch:
diff --git a/docs/commands/push.md b/docs/commands/push.md
index 3070e39b..c11066e5 100644
--- a/docs/commands/push.md
+++ b/docs/commands/push.md
@@ -89,6 +89,7 @@ destination      | string    | Conditional. The location in the API registry whe
 --skip-decorator | [string] | Ignore one or more decorators. See the [Skip decorator section](#skip-decorator) for usage examples.
 --upsert, -u | boolean | Upsert an API to the API registry. See [the Upsert an API with push section](#upsert-an-api-with-push) for more information.  |
 --version     | boolean | Show version number.  |
+--files|string[]|List of other folders and files to upload. See [the Files section](#files) for more information. |
 
 ## Examples
 
@@ -276,6 +277,28 @@ For more information on how to configure access to your APIs, check the [registr
 redocly push openapi/petstore.yaml @openapi-org/petstore-api@v1 --public
 ```
 
+### Files
+
+The `--files` option allows you to upload other folders and files.
+
+```bash
+redocly push openapi/petstore.yaml @openapi-org/petstore-api@v1 --files ./path/to/folder
+```
+
+You can also add files and folders providing them in `.redocly.yaml`:
+
+```yaml
+apis:
+  main: 
+     root: ./openapi.yam;
+files:
+  - ./path/to/folder
+  - ./path/to/another-folder/file.md
+  - ./file.md
+```
+
+**`--files` has bigger priority than `.redocly.yaml`**
+
 ### Set up CI from Redocly Workflows
 
 The Redocly Workflows interface can help you get started with the `push` command.
diff --git a/packages/cli/src/commands/bundle.ts b/packages/cli/src/commands/bundle.ts
index 7636d3e2..9bd1c768 100644
--- a/packages/cli/src/commands/bundle.ts
+++ b/packages/cli/src/commands/bundle.ts
@@ -34,7 +34,7 @@ export type BundleOptions = CommonOptions &
   };
 
 export async function handleBundle(argv: BundleOptions, version: string) {
-  const config = await loadConfig(argv.config, argv.extends);
+  const config = await loadConfig({ configPath: argv.config, customExtends: argv.extends });
   const removeUnusedComponents =
     argv['remove-unused-components'] &&
     !config.rawConfig.styleguide?.decorators?.hasOwnProperty('remove-unused-components');
diff --git a/packages/cli/src/commands/lint.ts b/packages/cli/src/commands/lint.ts
index 5ac1389b..4aa5df1f 100644
--- a/packages/cli/src/commands/lint.ts
+++ b/packages/cli/src/commands/lint.ts
@@ -39,11 +39,11 @@ export async function handleLint(argv: LintOptions, version: string) {
     return exitWithError('Please, provide valid path to the configuration file');
   }
 
-  const config: Config = await loadConfig(
-    argv.config,
-    argv.extends,
-    lintConfigCallback(argv, version)
-  );
+  const config: Config = await loadConfig({
+    configPath: argv.config,
+    customExtends: argv.extends,
+    processRawConfig: lintConfigCallback(argv, version),
+  });
 
   const apis = await getFallbackApisOrExit(argv.apis, config);
 
diff --git a/packages/cli/src/commands/preview-docs/index.ts b/packages/cli/src/commands/preview-docs/index.ts
index 008b5d45..cdc81374 100644
--- a/packages/cli/src/commands/preview-docs/index.ts
+++ b/packages/cli/src/commands/preview-docs/index.ts
@@ -129,7 +129,7 @@ export async function previewDocs(
   });
 
   async function reloadConfig() {
-    const config = await loadConfig(argv.config);
+    const config = await loadConfig({ configPath: argv.config });
     const redoclyClient = new RedoclyClient();
     isAuthorizedWithRedocly = await redoclyClient.isAuthorizedWithRedocly();
     const resolvedConfig = getMergedConfig(config, argv.api);
diff --git a/packages/cli/src/commands/push.ts b/packages/cli/src/commands/push.ts
index 765e2be3..2de37781 100644
--- a/packages/cli/src/commands/push.ts
+++ b/packages/cli/src/commands/push.ts
@@ -37,12 +37,12 @@ type PushArgs = {
   region?: Region;
   'skip-decorator'?: string[];
   public?: boolean;
+  files?: string[];
 };
 
 export async function handlePush(argv: PushArgs): Promise<void> {
-  const config = await loadConfig();
-  const region = argv.region || config.region;
-  const client = new RedoclyClient(region);
+  const config = await loadConfig({ region: argv.region, files: argv.files });
+  const client = new RedoclyClient(config.region);
   const isAuthorized = await client.isAuthorizedWithRedoclyByRegion();
   if (!isAuthorized) {
     const clientToken = await promptClientToken(client.domain);
@@ -254,6 +254,20 @@ async function collectFilesToUpload(api: string, config: Config) {
     }
     files.push(...filterPluginFilesByExt(Array.from(pluginFiles)).map((f) => getFileEntry(f)));
   }
+
+  if (config.files) {
+    const otherFiles = new Set<string>();
+    for (const file of config.files) {
+      if (fs.statSync(file).isDirectory()) {
+        const fileList = getFilesList(file, []);
+        fileList.forEach((f) => otherFiles.add(f));
+      } else {
+        otherFiles.add(file);
+      }
+    }
+    files.push(...Array.from(otherFiles).map((f) => getFileEntry(f)));
+  }
+
   return {
     files,
     root: path.resolve(apiPath),
diff --git a/packages/cli/src/commands/stats.ts b/packages/cli/src/commands/stats.ts
index 608a0acb..5724e343 100755
--- a/packages/cli/src/commands/stats.ts
+++ b/packages/cli/src/commands/stats.ts
@@ -66,7 +66,7 @@ function printStats(statsAccumulator: StatsAccumulator, api: string, format: str
 }
 
 export async function handleStats(argv: { config?: string; api?: string; format: string }) {
-  const config: Config = await loadConfig(argv.config);
+  const config: Config = await loadConfig({ configPath: argv.config });
   const [{ path }] = await getFallbackApisOrExit(argv.api ? [argv.api] : [], config);
   const externalRefResolver = new BaseResolver(config.resolve);
   const { bundle: document } = await bundle({ config, ref: path });
diff --git a/packages/cli/src/index.ts b/packages/cli/src/index.ts
index 0ee0a6b6..7cac9aa6 100644
--- a/packages/cli/src/index.ts
+++ b/packages/cli/src/index.ts
@@ -131,6 +131,11 @@ yargs
             description: 'Make API registry available to the public',
             type: 'boolean',
           },
+          files: {
+            description: 'List of other folders and files to upload',
+            array: true,
+            type: 'string',
+          },
         })
         .implies('batch-id', 'batch-size')
         .implies('batch-size', 'batch-id'),
diff --git a/packages/core/README.md b/packages/core/README.md
index fc7f3404..75d4900b 100644
--- a/packages/core/README.md
+++ b/packages/core/README.md
@@ -10,7 +10,7 @@ See https://github.com/Redocly/redocly-cli
 import { formatProblems, lint, loadConfig } from '@redocly/openapi-core';
 
 const pathToApi = 'openapi.yaml';
-const config = loadConfig('optional/path/to/.redocly.yaml');
+const config = loadConfig({ configPath: 'optional/path/to/.redocly.yaml' });
 const lintResults = await lint({ ref: pathToApi, config });
 ```
 
@@ -20,6 +20,6 @@ const lintResults = await lint({ ref: pathToApi, config });
 import { formatProblems, bundle, loadConfig } from '@redocly/openapi-core';
 
 const pathToApi = 'openapi.yaml';
-const config = loadConfig('optional/path/to/.redocly.yaml');
+const config = loadConfig({ configPath: 'optional/path/to/.redocly.yaml' });
 const { bundle, problems } = await bundle({ ref: pathToApi, config });
 ```
diff --git a/packages/core/src/config/config.ts b/packages/core/src/config/config.ts
index 34589f0e..5714c251 100755
--- a/packages/core/src/config/config.ts
+++ b/packages/core/src/config/config.ts
@@ -306,6 +306,7 @@ export class Config {
   'features.openapi': Record<string, any>;
   'features.mockServer'?: Record<string, any>;
   organization?: string;
+  files: string[];
   constructor(public rawConfig: ResolvedConfig, public configFile?: string) {
     this.apis = rawConfig.apis || {};
     this.styleguide = new StyleguideConfig(rawConfig.styleguide || {}, configFile);
@@ -314,5 +315,6 @@ export class Config {
     this.resolve = getResolveConfig(rawConfig?.resolve);
     this.region = rawConfig.region;
     this.organization = rawConfig.organization;
+    this.files = rawConfig.files || [];
   }
 }
diff --git a/packages/core/src/config/load.ts b/packages/core/src/config/load.ts
index cd632ac4..2e871198 100644
--- a/packages/core/src/config/load.ts
+++ b/packages/core/src/config/load.ts
@@ -62,11 +62,17 @@ async function addConfigMetadata({
 }
 
 export async function loadConfig(
-  configPath: string | undefined = findConfig(),
-  customExtends?: string[],
-  processRawConfig?: (rawConfig: RawConfig) => void | Promise<void>
+  options: {
+    configPath?: string;
+    customExtends?: string[];
+    processRawConfig?: (rawConfig: RawConfig) => void | Promise<void>;
+    files?: string[];
+    region?: Region;
+  } = {}
 ): Promise<Config> {
-  const rawConfig = await getConfig(configPath);
+  const { configPath = findConfig(), customExtends, processRawConfig, files, region } = options;
+  const config = await getConfig(configPath);
+  const rawConfig = { ...config, files: files ?? config.files, region: region ?? config.region };
   if (typeof processRawConfig === 'function') {
     await processRawConfig(rawConfig);
   }
diff --git a/packages/core/src/config/types.ts b/packages/core/src/config/types.ts
index 69ecc769..bed8ca75 100644
--- a/packages/core/src/config/types.ts
+++ b/packages/core/src/config/types.ts
@@ -145,6 +145,7 @@ export type DeprecatedInApi = {
 
 export type ResolvedApi = Omit<Api, 'styleguide'> & {
   styleguide: ResolvedStyleguideConfig;
+  files?: string[];
 };
 
 export type RawConfig = {
@@ -153,6 +154,7 @@ export type RawConfig = {
   resolve?: RawResolveConfig;
   region?: Region;
   organization?: string;
+  files?: string[];
 } & FeaturesConfig;
 
 export type FlatApi = Omit<Api, 'styleguide'> &
diff --git a/packages/core/src/config/utils.ts b/packages/core/src/config/utils.ts
index 49254bd6..8afd0b3d 100644
--- a/packages/core/src/config/utils.ts
+++ b/packages/core/src/config/utils.ts
@@ -233,6 +233,7 @@ export function getMergedConfig(config: Config, apiName?: string): Config {
             ...config['features.mockServer'],
             ...config.apis[apiName]?.['features.mockServer'],
           },
+          files: [...config.files, ...(config.apis?.[apiName]?.files ?? [])],
           // TODO: merge everything else here
         },
         config.configFile
diff --git a/packages/core/src/types/redocly-yaml.ts b/packages/core/src/types/redocly-yaml.ts
index 8c3481fd..a868c0c5 100644
--- a/packages/core/src/types/redocly-yaml.ts
+++ b/packages/core/src/types/redocly-yaml.ts
@@ -165,6 +165,12 @@ const ConfigRoot: NodeType = {
         doNotResolveExamples: { type: 'boolean' },
       },
     },
+    files: {
+      type: 'array',
+      items: {
+        type: 'string',
+      },
+    },
   },
 };
 
@@ -187,6 +193,12 @@ const ConfigApisProperties: NodeType = {
     ...ConfigStyleguide.properties,
     'features.openapi': 'ConfigReferenceDocs',
     'features.mockServer': 'ConfigMockServer',
+    files: {
+      type: 'array',
+      items: {
+        type: 'string',
+      },
+    },
   },
   required: ['root'],
 };


Test patch:
diff --git a/packages/cli/src/__tests__/commands/lint.test.ts b/packages/cli/src/__tests__/commands/lint.test.ts
index 66cfa0eb..859fd32c 100644
--- a/packages/cli/src/__tests__/commands/lint.test.ts
+++ b/packages/cli/src/__tests__/commands/lint.test.ts
@@ -63,7 +63,11 @@ describe('handleLint', () => {
 
     it('should call loadConfig and getFallbackApisOrExit', async () => {
       await handleLint(argvMock, versionMock);
-      expect(loadConfig).toHaveBeenCalledWith(undefined, undefined, undefined);
+      expect(loadConfig).toHaveBeenCalledWith({
+        configPath: undefined,
+        customExtends: undefined,
+        processRawConfig: undefined,
+      });
       expect(getFallbackApisOrExit).toHaveBeenCalled();
     });
 
@@ -72,7 +76,11 @@ describe('handleLint', () => {
         { ...argvMock, config: 'redocly.yaml', extends: ['some/path'] },
         versionMock
       );
-      expect(loadConfig).toHaveBeenCalledWith('redocly.yaml', ['some/path'], undefined);
+      expect(loadConfig).toHaveBeenCalledWith({
+        configPath: 'redocly.yaml',
+        customExtends: ['some/path'],
+        processRawConfig: undefined,
+      });
     });
 
     it('should call mergedConfig with clear ignore if `generate-ignore-file` argv', async () => {
diff --git a/packages/cli/src/__tests__/commands/push.test.ts b/packages/cli/src/__tests__/commands/push.test.ts
index a6c251ee..f6cc4c5a 100644
--- a/packages/cli/src/__tests__/commands/push.test.ts
+++ b/packages/cli/src/__tests__/commands/push.test.ts
@@ -1,6 +1,8 @@
-import { Config, getMergedConfig } from '@redocly/openapi-core';
+import * as fs from 'fs';
+import { Config, getMergedConfig, loadConfig } from '@redocly/openapi-core';
 import { exitWithError } from '../../utils';
 import { getApiRoot, getDestinationProps, handlePush, transformPush } from '../../commands/push';
+import { ConfigFixture } from '../fixtures/config';
 
 jest.mock('fs');
 jest.mock('node-fetch', () => ({
@@ -75,6 +77,36 @@ describe('push', () => {
 
     expect(exitWithError).toBeCalledTimes(1);
   });
+
+  it('push with --files', async () => {
+    (loadConfig as jest.Mock).mockImplementation(({ files }) => {
+      return { ...ConfigFixture, files };
+    });
+
+    //@ts-ignore
+    fs.statSync.mockImplementation(() => {
+      return { isDirectory: () => false, size: 10 };
+    });
+
+    await handlePush({
+      upsert: true,
+      api: 'spec.json',
+      destination: '@org/my-api@1.0.0',
+      public: true,
+      files: ['./resouces/1.md', './resouces/2.md'],
+    });
+
+    expect(redoclyClient.registryApi.pushApi).toHaveBeenLastCalledWith({
+      filePaths: ['filePath', 'filePath', 'filePath'],
+      isUpsert: true,
+      isPublic: true,
+      name: 'my-api',
+      organizationId: 'org',
+      rootFilePath: 'filePath',
+      version: '1.0.0',
+    });
+    expect(redoclyClient.registryApi.prepareFileUpload).toBeCalledTimes(3);
+  });
 });
 
 describe('transformPush', () => {
diff --git a/packages/core/src/config/__tests__/config.test.ts b/packages/core/src/config/__tests__/config.test.ts
index dc8a197e..22fc2bbe 100644
--- a/packages/core/src/config/__tests__/config.test.ts
+++ b/packages/core/src/config/__tests__/config.test.ts
@@ -48,6 +48,7 @@ const testConfig: Config = {
   'features.mockServer': {},
   resolve: { http: { headers: [] } },
   organization: 'redocly-test',
+  files: [],
 };
 
 describe('getMergedConfig', () => {
@@ -67,6 +68,7 @@ describe('getMergedConfig', () => {
         "configFile": "redocly.yaml",
         "features.mockServer": Object {},
         "features.openapi": Object {},
+        "files": Array [],
         "organization": "redocly-test",
         "rawConfig": Object {
           "apis": Object {
@@ -81,6 +83,7 @@ describe('getMergedConfig', () => {
           },
           "features.mockServer": Object {},
           "features.openapi": Object {},
+          "files": Array [],
           "organization": "redocly-test",
           "styleguide": Object {
             "extendPaths": Array [],
@@ -163,6 +166,7 @@ describe('getMergedConfig', () => {
         "configFile": "redocly.yaml",
         "features.mockServer": Object {},
         "features.openapi": Object {},
+        "files": Array [],
         "organization": "redocly-test",
         "rawConfig": Object {
           "apis": Object {
@@ -177,6 +181,7 @@ describe('getMergedConfig', () => {
           },
           "features.mockServer": Object {},
           "features.openapi": Object {},
+          "files": Array [],
           "organization": "redocly-test",
           "styleguide": Object {
             "extendPaths": Array [],
diff --git a/packages/core/src/config/__tests__/load.test.ts b/packages/core/src/config/__tests__/load.test.ts
index 15670ca2..632f49ba 100644
--- a/packages/core/src/config/__tests__/load.test.ts
+++ b/packages/core/src/config/__tests__/load.test.ts
@@ -49,7 +49,7 @@ describe('loadConfig', () => {
 
   it('should call callback if such passed', async () => {
     const mockFn = jest.fn();
-    await loadConfig(undefined, undefined, mockFn);
+    await loadConfig({ processRawConfig: mockFn });
     expect(mockFn).toHaveBeenCalled();
   });
 });
diff --git a/packages/core/src/config/__tests__/resolve-plugins.test.ts b/packages/core/src/config/__tests__/resolve-plugins.test.ts
index 48053701..badbf3f9 100644
--- a/packages/core/src/config/__tests__/resolve-plugins.test.ts
+++ b/packages/core/src/config/__tests__/resolve-plugins.test.ts
@@ -5,21 +5,21 @@ describe('resolving a plugin', () => {
   const configPath = path.join(__dirname, 'fixtures/plugin-config.yaml');
 
   it('should prefix rule names with the plugin id', async () => {
-    const config = await loadConfig(configPath);
+    const config = await loadConfig({ configPath });
     const plugin = config.styleguide.plugins[0];
 
     expect(plugin.rules?.oas3).toHaveProperty('test-plugin/openid-connect-url-well-known');
   });
 
   it('should prefix preprocessor names with the plugin id', async () => {
-    const config = await loadConfig(configPath);
+    const config = await loadConfig({ configPath });
     const plugin = config.styleguide.plugins[0];
 
     expect(plugin.preprocessors?.oas2).toHaveProperty('test-plugin/description-preprocessor');
   });
 
   it('should prefix decorator names with the plugin id', async () => {
-    const config = await loadConfig(configPath);
+    const config = await loadConfig({ configPath });
     const plugin = config.styleguide.plugins[0];
 
     expect(plugin.decorators?.oas3).toHaveProperty('test-plugin/inject-x-stats');