Repository: hyperjumptech/monika
Pull request #883 (2022-09-15 06:06:44)

feat-811/sitemap-one-probe
# Monika Pull Request (PR)  

## What feature/issue does this PR add  
1. add `--one-probe` options on sitemap (#811 )

## How did you implement / how did you fix it  
1.  run `/bin/run_dev --sitemap ./src/components/config/__tests__/sitemap.xml --one-probe`

## How to test  
1. test case was included (./src/components/config/__tests__/sitemap.xml) and expected result (./src/components/config/__tests__/expected.sitemap-oneprobe.yml)

<img width="971" alt="image" src="https://user-images.githubusercontent.com/42835003/190328868-bcd46495-c6f4-42a8-bf28-59d0ff90e1cd.png">



codecov[bot]: # [Codecov](https://codecov.io/gh/hyperjumptech/monika/pull/883?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=hyperjumptech) Report
> Merging [#883](https://codecov.io/gh/hyperjumptech/monika/pull/883?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=hyperjumptech) (c2bef63) into [main](https://codecov.io/gh/hyperjumptech/monika/commit/385fa6118b5b5417572b24f2d6d23f4a3ade96c8?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=hyperjumptech) (385fa61) will **increase** coverage by `0.21%`.
> The diff coverage is `80.95%`.

```diff
@@            Coverage Diff             @@
##             main     #883      +/-   ##
==========================================
+ Coverage   48.08%   48.29%   +0.21%     
==========================================
  Files          72       72              
  Lines        2475     2489      +14     
  Branches      494      498       +4     
==========================================
+ Hits         1190     1202      +12     
  Misses       1103     1103              
- Partials      182      184       +2     
```


| [Impacted Files](https://codecov.io/gh/hyperjumptech/monika/pull/883?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=hyperjumptech) | Coverage Δ | |
|---|---|---|
| [src/index.ts](https://codecov.io/gh/hyperjumptech/monika/pull/883/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=hyperjumptech#diff-c3JjL2luZGV4LnRz) | `54.21% <ø> (ø)` | |
| [src/components/config/index.ts](https://codecov.io/gh/hyperjumptech/monika/pull/883/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=hyperjumptech#diff-c3JjL2NvbXBvbmVudHMvY29uZmlnL2luZGV4LnRz) | `64.74% <50.00%> (ø)` | |
| [src/components/config/parse-sitemap.ts](https://codecov.io/gh/hyperjumptech/monika/pull/883/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=hyperjumptech#diff-c3JjL2NvbXBvbmVudHMvY29uZmlnL3BhcnNlLXNpdGVtYXAudHM=) | `76.47% <87.50%> (+6.47%)` | :arrow_up: |
| [src/components/config/parse.ts](https://codecov.io/gh/hyperjumptech/monika/pull/883/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=hyperjumptech#diff-c3JjL2NvbXBvbmVudHMvY29uZmlnL3BhcnNlLnRz) | `52.50% <100.00%> (ø)` | |

:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=hyperjumptech)


Patch:
diff --git a/docs/src/pages/guides/cli-options.md b/docs/src/pages/guides/cli-options.md
index c20acc66..cecd65c2 100644
--- a/docs/src/pages/guides/cli-options.md
+++ b/docs/src/pages/guides/cli-options.md
@@ -211,6 +211,12 @@ Monika supports reading sitemap.xml as configuration input. Use the `--sitemap`
 monika --sitemap sitemap.xml
 ```
 
+or you can also add `--one-probe` flag to merge all requests into single probe.
+
+```bash
+monika --sitemap sitemap.xml --one-probe
+```
+
 ### Create config from Sitemap file
 
 You can use the combination of `--create-config` and `--sitemap` flags to convert the sitemap xml file to a monika.yml config file.
@@ -219,6 +225,12 @@ You can use the combination of `--create-config` and `--sitemap` flags to conver
 monika --create-config --sitemap sitemap.xml
 ```
 
+or you can also add `--one-probe` flag to merge all requests into single probe.
+
+```bash
+monika --create-config --sitemap sitemap.xml --one-probe
+```
+
 ## Prometheus
 
 You can expose the [Prometheus](https://prometheus.io/) metrics server with the `--prometheus` flag and server port as a value.
diff --git a/src/components/config/index.ts b/src/components/config/index.ts
index 96b3c455..34fe72f6 100644
--- a/src/components/config/index.ts
+++ b/src/components/config/index.ts
@@ -107,7 +107,8 @@ const watchConfigFile = (
   path: string,
   type: string,
   index?: number,
-  repeat?: number
+  repeat?: number,
+  flags?: any
 ) => {
   const watchConfigFile = !(
     process.env.CI ||
@@ -117,7 +118,7 @@ const watchConfigFile = (
   if (watchConfigFile) {
     const watcher = chokidar.watch(path)
     watcher.on('change', async () => {
-      const newConfig = await parseConfig(path, type)
+      const newConfig = await parseConfig(path, type, flags)
       if (index === undefined) {
         nonDefaultConfig = newConfig
       } else {
@@ -133,11 +134,12 @@ const scheduleRemoteConfigFetcher = (
   url: string,
   configType: 'monika' | 'har' | 'insomnia' | 'postman' | 'sitemap',
   interval: number,
-  index?: number
+  index?: number,
+  flags?: any
 ) => {
   setInterval(async () => {
     try {
-      const newConfig = await parseConfig(url, configType)
+      const newConfig = await parseConfig(url, configType, flags)
       if (index === undefined) {
         nonDefaultConfig = newConfig
       } else {
@@ -164,7 +166,7 @@ const parseConfigType = async (
     watchConfigFile(source, configType, index, flags.repeat)
   }
 
-  const parsed = await parseConfig(source, configType)
+  const parsed = await parseConfig(source, configType, flags)
 
   return {
     ...parsed,
@@ -282,7 +284,7 @@ export const createConfig = async (flags: any): Promise<void> => {
       return
     }
 
-    const parse = await parseConfig(path, type)
+    const parse = await parseConfig(path, type, flags)
     const result = await addDefaultNotifications(parse)
     const file = flags.output || 'monika.yml'
 
diff --git a/src/components/config/parse-sitemap.ts b/src/components/config/parse-sitemap.ts
index 19beace1..a470c2a0 100644
--- a/src/components/config/parse-sitemap.ts
+++ b/src/components/config/parse-sitemap.ts
@@ -25,6 +25,8 @@
 import { Config } from '../../interfaces/config'
 import { DEFAULT_THRESHOLD } from '../../looper'
 import { XMLParser } from 'fast-xml-parser'
+import { Probe } from '../../interfaces/probe'
+import { DEFAULT_CONFIG_INTERVAL } from '.'
 
 const generateProbesFromXml = (config: any) => {
   const probes = config?.urlset?.url?.map((item: any) => {
@@ -58,12 +60,70 @@ const generateProbesFromXml = (config: any) => {
   return probes ?? []
 }
 
-export const parseConfigFromSitemap = (configString: string): Config => {
+const generateProbesFromXmlOneProbe = (config: any) => {
+  let probe: Probe | undefined
+  let requests: any = []
+
+  const resources = config?.urlset?.url
+  for (const [, item] of resources.entries()) {
+    // eslint-disable-next-line prefer-const
+    requests = [
+      ...requests,
+      {
+        url: item.loc,
+        method: 'GET',
+        timeout: 10_000,
+      },
+    ]
+    if (item['xhtml:link']) {
+      for (const alt of item['xhtml:link']) {
+        requests = [
+          ...requests,
+          {
+            url: alt['@_href'],
+            method: 'GET',
+            timeout: 10_000,
+          },
+        ]
+      }
+    }
+
+    const url = new URL(item.loc)
+
+    probe = {
+      id: url.host,
+      name: url.host,
+      requests: requests,
+      interval: DEFAULT_CONFIG_INTERVAL,
+      incidentThreshold: DEFAULT_THRESHOLD,
+      recoveryThreshold: DEFAULT_THRESHOLD,
+      alerts: [
+        {
+          query: 'response.status < 200 or response.status > 299',
+          message: 'HTTP Status is not 200',
+        },
+        {
+          query: 'response.time > 2000',
+          message: 'Response time is more than 2000ms',
+        },
+      ],
+    }
+  }
+
+  return probe ? [probe] : []
+}
+
+export const parseConfigFromSitemap = (
+  configString: string,
+  flags: any
+): Config => {
   try {
     const parser = new XMLParser({ ignoreAttributes: false })
     const xmlObj = parser.parse(configString)
-    const probes = generateProbesFromXml(xmlObj)
-
+    let probes = generateProbesFromXml(xmlObj)
+    if (flags && flags['one-probe']) {
+      probes = generateProbesFromXmlOneProbe(xmlObj)
+    }
     return { probes }
   } catch (error: any) {
     throw new Error(
diff --git a/src/components/config/parse.ts b/src/components/config/parse.ts
index b3e459ae..f43c44a3 100644
--- a/src/components/config/parse.ts
+++ b/src/components/config/parse.ts
@@ -40,7 +40,8 @@ function sleep(ms: number): Promise<void> {
 
 export const parseConfig = async (
   source: string,
-  type: string
+  type: string,
+  flag: any
 ): Promise<Partial<Config>> => {
   try {
     let configString = isUrl(source)
@@ -73,7 +74,7 @@ export const parseConfig = async (
 
     if (type === 'har') return parseHarFile(configString)
     if (type === 'postman') return parseConfigFromPostman(configString)
-    if (type === 'sitemap') return parseConfigFromSitemap(configString)
+    if (type === 'sitemap') return parseConfigFromSitemap(configString, flag)
     if (type === 'insomnia')
       return parseInsomnia(configString, ext.replace('.', ''))
 
diff --git a/src/index.ts b/src/index.ts
index a91e758d..c23eed6e 100644
--- a/src/index.ts
+++ b/src/index.ts
@@ -157,6 +157,11 @@ class Monika extends Command {
       exclusive: ['har', 'insomnia', 'postman'],
     }),
 
+    'one-probe': Flags.boolean({
+      description: 'One Probe',
+      dependsOn: ['sitemap'],
+    }),
+
     postman: Flags.string({
       char: 'p', // (p)ostman
       description: 'Run Monika using a Postman json file.',
@@ -496,7 +501,7 @@ Please refer to the Monika documentations on how to how to configure notificatio
         }
         startupMessage += `    Alerts: ${
           probe?.alerts === undefined || probe?.alerts.length === 0
-            ? `[{ "query": "response.status < 200 or response.status > 299", "message": "HTTP Status is not 200"}, 
+            ? `[{ "query": "response.status < 200 or response.status > 299", "message": "HTTP Status is not 200"},
             { "query": "response.time > 2000", "message": "Response time is more than 2000ms" }]`
             : JSON.stringify(probe.alerts)
         }\n`


Test patch:
diff --git a/src/components/config/__tests__/create-config.test.ts b/src/components/config/__tests__/create-config.test.ts
index 00c92c31..aa6b3376 100644
--- a/src/components/config/__tests__/create-config.test.ts
+++ b/src/components/config/__tests__/create-config.test.ts
@@ -94,6 +94,27 @@ describe('Sitemap config', () => {
   })
 })
 
+describe('Sitemap config [--one-probe ]', () => {
+  describe('Create config from xml file', () => {
+    it('should create config from xml file', async () => {
+      const flags = {
+        sitemap: './src/components/config/__tests__/sitemap.xml',
+        output: 'monika.sitemap.yml',
+        'one-probe': true,
+      }
+      await createConfig(flags)
+      expect(fs.lstatSync('monika.sitemap.yml').isFile()).to.be.true
+
+      const generated = fs.readFileSync('monika.sitemap.yml', 'utf-8')
+      const expected = fs.readFileSync(
+        './src/components/config/__tests__/expected.sitemap-oneprobe.yml',
+        'utf-8'
+      )
+      expect(_.isEqual(generated, expected)).to.be.true
+    })
+  })
+})
+
 const getPostmanConfig = ({ grouped }: { grouped: boolean }) => {
   if (grouped) {
     const generated = fs.readFileSync('monika.postman-grouped.yml', 'utf-8')
diff --git a/src/components/config/__tests__/expected.sitemap-oneprobe.yml b/src/components/config/__tests__/expected.sitemap-oneprobe.yml
new file mode 100644
index 00000000..5ebac108
--- /dev/null
+++ b/src/components/config/__tests__/expected.sitemap-oneprobe.yml
@@ -0,0 +1,30 @@
+probes:
+  - id: monika.hyperjump.tech
+    name: monika.hyperjump.tech
+    requests:
+      - url: https://monika.hyperjump.tech/articles
+        method: GET
+        timeout: 10000
+      - url: https://de.wiktionary.org/wiki/hyperjump
+        method: GET
+        timeout: 10000
+      - url: https://id.wiktionary.org/wiki/hyperjump
+        method: GET
+        timeout: 10000
+      - url: https://en.wiktionary.org/wiki/hyperjump
+        method: GET
+        timeout: 10000
+      - url: https://monika.hyperjump.tech/index
+        method: GET
+        timeout: 10000
+    interval: 900
+    incidentThreshold: 5
+    recoveryThreshold: 5
+    alerts:
+      - query: response.status < 200 or response.status > 299
+        message: HTTP Status is not 200
+      - query: response.time > 2000
+        message: Response time is more than 2000ms
+notifications:
+  - id: default
+    type: desktop