Repository: garmeeh/next-seo
Pull request #581 (2020-11-13 22:17:11)

[Feature] robotsProps (An Update for robots meta tags)
## Description of Change(s):

Update for the new `robots` meta tags values defined by [Google Search Central - Control Crawling and indexing](https://developers.google.com/search/reference/robots_meta_tag?hl=en-GB#directives).

Instead of using `additionalMetaTags`, that duplicates the meta tag and overrides it, I introduced the new prop `robotsProps` to the `NextSEO` component, this solves the complex integration of new meta values for crawling introduced by Google.

An example here:

```jsx
const Page = () => (
  <>
    <NextSeo
      robotsProps={{
        nosnippet: true,
        notranslate: true,
        noimageindex: true,
        noarchive: true,
        maxSnippet: -1,
        maxImagePreview: 'none',
        maxVideoPreview: -1,
      }}
    />
    <p>Additional robots props in Next-SEO!!</p>
  </>
);

export default Page;
```

### What's new

- `robotsProps` Prop.
- Fully typed.
- `/robots` page to test the additional props.
- Unit test cases for `robotsProps` for `index,follow` / `noindex,nofollow`.
- Integration tests for the new `/robots` page and its `<head>`.
- Documentation section.

---

Some things to help get a PR reviewed and merged faster:

- Have you updated the documentation to go with your changes? **Yes, `#robotsProps`**
- Have you written or updated unit tests? **Yes, I added tests for new meta cases**
- Have you written an integration test in the test app supplied? **Yes, added an integration test for the `robots.tsx` page**

rodzy: This solves (#521), @garmeeh let me know what you think about this change, thanks as always 😄 
rodzy: @garmeeh Changes were quickly applied, I'll be happy to keep helping with future issues 👍 
garmeeh: @rodzy If you would like to contribute some more, I would be more than happy to outline the changes I was thinking of making and maybe you would be interested in making them? 

Patch:
diff --git a/README.md b/README.md
index 2acc4d1..265dfd7 100644
--- a/README.md
+++ b/README.md
@@ -28,6 +28,7 @@ looking for inspiration on what to add.
     - [dangerouslySetAllPagesToNoIndex](#dangerouslysetallpagestonoindex)
     - [No Follow](#no-follow)
     - [dangerouslySetAllPagesToNoFollow](#dangerouslysetallpagestonofollow)
+    - [robotsProps](#robotsprops)
     - [Twitter](#twitter)
     - [facebook](#facebook)
     - [Canonical URL](#canonical-url)
@@ -41,7 +42,7 @@ looking for inspiration on what to add.
     - [Book](#book)
     - [Profile](#profile)
 - [JSON-LD](#json-ld)
-  - [Handling multiple instances...](#handling-multiple-instances)
+  - [Handling multiple instances](#handling-multiple-instances)
   - [Article](#article-1)
   - [Breadcrumb](#breadcrumb)
   - [Blog](#blog)
@@ -250,6 +251,7 @@ From now on all of your pages will have the defaults above applied.
 | `defaultTitle`                     | string                  | If no title is set on a page, this string will be used instead of an empty `titleTemplate` [More Info](#default-title)                                                               |
 | `noindex`                          | boolean (default false) | Sets whether page should be indexed or not [More Info](#no-index)                                                                                                                    |
 | `nofollow`                         | boolean (default false) | Sets whether page should be followed or not [More Info](#no-follow)                                                                                                                  |
+| `additionRobotsProps`              | Object                  | Set the more meta information for the `X-Robots-Tag` [More Info](#additionalrobotsprops)                                                                                             |
 | `description`                      | string                  | Set the page meta description                                                                                                                                                        |
 | `canonical`                        | string                  | Set the page canonical url                                                                                                                                                           |
 | `mobileAlternate.media`            | string                  | Set what screen size the mobile website should be served from                                                                                                                        |
@@ -386,6 +388,55 @@ The only way to unset this, is by removing the prop from the `DefaultSeo` in you
 | false     | true       | `index,nofollow`                        |
 | true      | true       | `noindex,nofollow`                      |
 
+#### robotsProps
+
+In addition to `index, follow` the `robots` meta tag accepts more properties to archive a more accurate crawling and serve better snippets for SEO bots that crawl your page.
+
+Example:
+
+```jsx
+import { NextSeo } from 'next-seo';
+
+const Page = () => (
+  <>
+    <NextSeo
+      robotsProps={{
+        nosnippet: true,
+        notranslate: true,
+        noimageindex: true,
+        noarchive: true,
+        maxSnippet: -1,
+        maxImagePreview: 'none',
+        maxVideoPreview: -1,
+      }}
+    />
+    <p>Additional robots props in Next-SEO!!</p>
+  </>
+);
+
+export default Page;
+
+/*
+<meta name="robots" content="index,follow,nosnippet,max-snippet:-1,max-image-preview:none,noarchive,noimageindex,max-video-preview:-1,notranslate">
+<meta name="googlebot" content="index,follow,nosnippet,max-snippet:-1,max-image-preview:none,noarchive,noimageindex,max-video-preview:-1,notranslate">
+*/
+```
+
+**Available properties**
+
+| Property            | Type                      | Description                                                                                                                                                                                     |
+| ------------------- | ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `noarchive`         | boolean                   | Do not show a [cached link](https://support.google.com/websearch/answer/1687222) in search results.                                                                                             |
+| `nosnippet`         | boolean                   | Do not show a text snippet or video preview in the search results for this page.                                                                                                                |
+| `max-snippet`       | number                    | Use a maximum of [number] characters as a textual snippet for this search result. [Read more](https://developers.google.com/search/reference/robots_meta_tag?hl=en-GB#directives)               |
+| `max-image-preview` | 'none','standard','large' | Set the maximum size of an image preview for this page in a search results.                                                                                                                     |
+| `max-video-preview` | number                    | Use a maximum of [number] seconds as a video snippet for videos on this page in search results. [Read more](https://developers.google.com/search/reference/robots_meta_tag?hl=en-GB#directives) |
+| `notranslate`       | boolean                   | Do not offer translation of this page in search results.                                                                                                                                        |
+| `noimageindex`      | boolean                   | Do not index images on this page.                                                                                                                                                               |
+| `unavailable_after` | string                    | Do not show this page in search results after the specified date/time. The date/time must be specified in a widely adopted format including, but not limited to RFC 822, RFC 850, and ISO 8601. |
+
+For more reference about the `X-Robots-Tag` visit [Google Search Central - Control Crawling and Indexing](https://developers.google.com/search/reference/robots_meta_tag?hl=en-GB#directives)
+
 #### Twitter
 
 Twitter will read the `og:title`, `og:image` and `og:description` tags for their card, this is why `next-seo` omits `twitter:title`, `twitter:image` and `twitter:description` so not to duplicate.
@@ -862,7 +913,7 @@ Below you will find a very basic page implementing each of the available JSON-LD
 
 Pull request very welcome to add any from the list [found on here](https://developers.google.com/search/docs/data-types/article)
 
-#### Handling multiple instances...
+#### Handling multiple instances
 
 If your page requires multiple instances of a given JSON-LD component, you can specify unique `keyOverride` properties, and `next-seo` will handle the rest.
 
diff --git a/src/meta/buildTags.tsx b/src/meta/buildTags.tsx
index 712389c..3ba5251 100644
--- a/src/meta/buildTags.tsx
+++ b/src/meta/buildTags.tsx
@@ -41,6 +41,30 @@ const buildTags = (config: BuildTagsParams) => {
     defaults.nofollow ||
     config.dangerouslySetAllPagesToNoFollow;
 
+  let robotsParams = '';
+  if (config.robotsProps) {
+    const {
+      nosnippet,
+      maxSnippet,
+      maxImagePreview,
+      maxVideoPreview,
+      noarchive,
+      noimageindex,
+      notranslate,
+      unavailableAfter,
+    } = config.robotsProps;
+
+    robotsParams = `${nosnippet ? ',nosnippet' : ''}${
+      maxSnippet ? `,max-snippet:${maxSnippet}` : ''
+    }${maxImagePreview ? `,max-image-preview:${maxImagePreview}` : ''}${
+      noarchive ? ',noarchive' : ''
+    }${unavailableAfter ? `,unavailable_after:${unavailableAfter}` : ''}${
+      noimageindex ? ',noimageindex' : ''
+    }${maxVideoPreview ? `,max-video-preview:${maxVideoPreview}` : ''}${
+      notranslate ? ',notranslate' : ''
+    }`;
+  }
+
   if (noindex || nofollow) {
     if (config.dangerouslySetAllPagesToNoIndex) {
       defaults.noindex = true;
@@ -55,7 +79,7 @@ const buildTags = (config: BuildTagsParams) => {
         name="robots"
         content={`${noindex ? 'noindex' : 'index'},${
           nofollow ? 'nofollow' : 'follow'
-        }`}
+        }${robotsParams}`}
       />,
     );
     tagsToRender.push(
@@ -64,15 +88,23 @@ const buildTags = (config: BuildTagsParams) => {
         name="googlebot"
         content={`${noindex ? 'noindex' : 'index'},${
           nofollow ? 'nofollow' : 'follow'
-        }`}
+        }${robotsParams}`}
       />,
     );
   } else {
     tagsToRender.push(
-      <meta key="robots" name="robots" content="index,follow" />,
+      <meta
+        key="robots"
+        name="robots"
+        content={`index,follow${robotsParams}`}
+      />,
     );
     tagsToRender.push(
-      <meta key="googlebot" name="googlebot" content="index,follow" />,
+      <meta
+        key="googlebot"
+        name="googlebot"
+        content={`index,follow${robotsParams}`}
+      />,
     );
   }
 
diff --git a/src/meta/nextSEO.tsx b/src/meta/nextSEO.tsx
index 6908715..b8bf2f9 100644
--- a/src/meta/nextSEO.tsx
+++ b/src/meta/nextSEO.tsx
@@ -9,6 +9,7 @@ export default class extends Component<NextSeoProps, {}> {
       title,
       noindex = false,
       nofollow,
+      robotsProps,
       description,
       canonical,
       openGraph,
@@ -26,6 +27,7 @@ export default class extends Component<NextSeoProps, {}> {
           title,
           noindex,
           nofollow,
+          robotsProps,
           description,
           canonical,
           facebook,
diff --git a/src/types.ts b/src/types.ts
index c5ad1e9..aec0b5f 100644
--- a/src/types.ts
+++ b/src/types.ts
@@ -149,12 +149,26 @@ export interface HTTPEquivMetaTag extends BaseMetaTag {
 
 export type MetaTag = HTML5MetaTag | RDFaMetaTag | HTTPEquivMetaTag;
 
+export type ImagePrevSize = 'none' | 'standard' | 'large';
+
+export interface AdditionalRobotsProps {
+  nosnippet?: boolean;
+  maxSnippet?: number;
+  maxImagePreview?: ImagePrevSize;
+  maxVideoPreview?: number;
+  noarchive?: boolean;
+  unavailableAfter?: string;
+  noimageindex?: boolean;
+  notranslate?: boolean;
+}
+
 export interface NextSeoProps {
   title?: string;
   titleTemplate?: string;
   defaultTitle?: string;
   noindex?: boolean;
   nofollow?: boolean;
+  robotsProps?: AdditionalRobotsProps;
   description?: string;
   canonical?: string;
   mobileAlternate?: MobileAlternate;


Test patch:
diff --git a/cypress/e2e/seo.spec.js b/cypress/e2e/seo.spec.js
index 57edc2d..c1277d0 100644
--- a/cypress/e2e/seo.spec.js
+++ b/cypress/e2e/seo.spec.js
@@ -102,6 +102,22 @@ describe('SEO Meta', () => {
     );
   });
 
+  it('SEO Robots props applied correctly', () => {
+    cy.visit('http://localhost:3000/robots');
+    cy.get('h1').should('contain', 'Robots meta properties');
+    cy.get('head title').should('contain', 'Robots meta title');
+    cy.get('head meta[name="robots"]').should(
+      'have.attr',
+      'content',
+      'index,follow,nosnippet,max-snippet:-1,max-image-preview:none,noarchive,noimageindex,max-video-preview:-1,notranslate',
+    );
+    cy.get('head meta[name="googlebot"]').should(
+      'have.attr',
+      'content',
+      'index,follow,nosnippet,max-snippet:-1,max-image-preview:none,noarchive,noimageindex,max-video-preview:-1,notranslate',
+    );
+  });
+
   it('SEO overrides apply correctly', () => {
     cy.visit('http://localhost:3000/overridden');
     cy.get('h1').should('contain', 'Overridden Seo');
diff --git a/e2e/components/links.tsx b/e2e/components/links.tsx
index 0b76e7a..44c6e89 100644
--- a/e2e/components/links.tsx
+++ b/e2e/components/links.tsx
@@ -8,6 +8,11 @@ const Links = () => (
         <a>Default SEO</a>
       </Link>
     </li>
+    <li>
+      <Link href="/robots">
+        <a>All Robots props</a>
+      </Link>
+    </li>
     <li>
       <Link href="/article">
         <a>Article SEO</a>
diff --git a/e2e/pages/robots.tsx b/e2e/pages/robots.tsx
new file mode 100644
index 0000000..d73acc0
--- /dev/null
+++ b/e2e/pages/robots.tsx
@@ -0,0 +1,22 @@
+import React from 'react';
+import { NextSeo } from '../..';
+import Links from '../components/links';
+
+export default () => (
+  <>
+    <NextSeo
+      title="Robots meta title"
+      robotsProps={{
+        nosnippet: true,
+        notranslate: true,
+        noimageindex: true,
+        noarchive: true,
+        maxSnippet: -1,
+        maxImagePreview: 'none',
+        maxVideoPreview: -1,
+      }}
+    />
+    <h1>Robots meta properties</h1>
+    <Links />
+  </>
+);
diff --git a/src/meta/__tests__/buildTags.spec.tsx b/src/meta/__tests__/buildTags.spec.tsx
index 8f616ca..f0d3aba 100644
--- a/src/meta/__tests__/buildTags.spec.tsx
+++ b/src/meta/__tests__/buildTags.spec.tsx
@@ -1,6 +1,6 @@
 import React from 'react';
 import { getByText, cleanup, render } from 'react-testing-library';
-import { BuildTagsParams } from '../../types';
+import { BuildTagsParams, ImagePrevSize } from '../../types';
 
 import buildTags from '../buildTags';
 
@@ -852,3 +852,31 @@ it('correctly read noindex & nofollow false', () => {
   expect(Array.from(indexfollow).length).toBe(0);
   expect(Array.from(noindexnofollow).length).toBe(2);
 });
+
+it('correctly read all robots props', () => {
+  const overrideProps = {
+    ...SEO,
+    noindex: true,
+    nofollow: true,
+    robotsProps: {
+      nosnippet: true,
+      notranslate: true,
+      noimageindex: true,
+      noarchive: true,
+      maxSnippet: -1,
+      maxImagePreview: 'none' as ImagePrevSize,
+      maxVideoPreview: -1,
+    },
+  };
+  const tags = buildTags(overrideProps);
+  const { container } = render(<>{React.Children.toArray(tags)}</>);
+  const content = container.querySelectorAll(
+    'meta[content="index,follow,nosnippet,max-snippet:-1,max-image-preview:none,noarchive,noimageindex,max-video-preview:-1,notranslate"]',
+  );
+
+  const contentOverride = container.querySelectorAll(
+    'meta[content="noindex,nofollow,nosnippet,max-snippet:-1,max-image-preview:none,noarchive,noimageindex,max-video-preview:-1,notranslate"]',
+  );
+  expect(Array.from(content).length).toBe(0);
+  expect(Array.from(contentOverride).length).toBe(2);
+});