Amazon CloudSearch For Dummies
By: Kiem Ngoc To
Tags: Mti Technology, Amazon Cloudsearch, Database Searching, Cloudsearch, Mti

I. A Brief Introduction to Database Searching

Nowadays, there are so much information on the internet which may overwhelm the users. It’s hard to find exactly what they need. Let’s face it. Users are lazy. They are not going to browse through your entire websites or web apps for what they need. The harder it is to find what they need, the faster they leave. In other words, it matters to make your content search-friendly because that’s how you get more user. More user means more money to your pay cheque. This leads us to another problem.

Making searching easy, fast and accurate has traditionally required a lot of hard work. You have to:

Set up servers

Install configured search software

Write a bunch of code to integrate your search functionality into your app.

This complexity slows the development of new apps, and there is an extra layer of infrastructure that needs to be managed, scaled, tuned and upgraded. One of the most common challenges for search is trying to determine the optimal amount of hardware you need to meet the demand for both the data indexing and the volume of traffic to your site. Getting this wrong could lead to excess capacity. As a result, it would cost you extra money and resources, or worse: not meeting the demand of your costumers’ search requests, leaving them with slow or broken search functionality.

However, what if you can replace all of this complex search engine hardware and software infrastructure with a simple cloud-based search service that can manage everything for you?

II. What is Amazon CloudSearch

Amazon CloudSearch is an automatically scaling, cloud-based search service. It can be integrated into other applications. It offers:

Fully managed and fully- featured search service.

Automatically scales for data & traffic.

Handling both structured and unstructured data. Structured data is comprised of clearly defined data types whose pattern make the easily searchable such as: names, dates, address, credit card number etc. Unstructured data is comprised of data that is not easily searchable, including formats like audio, video and social media posting.

is comprised of clearly defined data types whose pattern make the easily searchable such as: names, dates, address, credit card number etc. is comprised of data that is not easily searchable, including formats like audio, video and social media posting. Near real-time indexing.

Up and running in less than 1 hour.

III. How Search Works & What Cloudsearch can offer

Let’s take a look at the image below, for the sake of simplicity.

What we have above is the representation of the search result for the keyword eyelash. On the right side is the relevance/ranking of our search result. On the left side we have the faceting for our search. Facets are a way to add specific, relevant options to search result pages or category listing, so that when users search for a product or browse categories they can see where in your catalogue they have ended up.

Filters use certain product data as visible criteria visitors can use to refine the list of product. On the other hand, facets take it one step further and allow users to narrow results by several dimensions simultaneously such as: range searching, boolean searching, field searching. As a result, users will have a more complex relevance for what they’re looking for.

To make it easy for developers, Amazon CloudSearch offers full-featured search:

Free Text (about 34 languages, plus “multiple” to handle mixed language fields), structured data and boolean search

faceting

Customizable relevance ranking

Fielded and range search

result sorting

Text Processing options

Near real-time indexing

IV. Amazon CloudSearch Architecture​

You interact with Amazon CloudSearch through three services:

Configuration service: The configuration service enables you to create and configure search domains. To set up a search domain, you give it a unique name and configure indexing options, text analysis schemes, availability options, scaling options, suggesters, and expressions

The configuration service enables you to create and configure search domains. To set up a search domain, you give it a unique name and configure indexing options, text analysis schemes, availability options, scaling options, suggesters, and expressions Document service: You use the document service to make changes to a domain’s searchable data. Each domain has a unique document service HTTP endpoint.

You use the document service to make changes to a domain’s searchable data. Each domain has a unique document service HTTP endpoint. Search service: The search service handles search and suggestion requests for a domain. Each domain has a unique search HTTP endpoint. When you send a search or suggest request, the search service returns a list of matching documents. Results can be returned in either JSON or XML.

V. Understanding Amazon CloudSearch Limits

Although Amazon CloudSearch is a well managed search engine, it also has limits.

VI. Amazon CloudSearch Integration

AWS Lambda natively supports Java, Go, PowerShell, Node.js, C#, Python, and Ruby code, and provides a Runtime API which allows you to use any additional programming languages to author your functions.

There are 3 main APIs:

1. Search:

Retrieves a list of documents that match the specified search criteria. How you specify the search criteria depends on which query parser you use. Amazon CloudSearch supports four query parsers:

simple : search all text and text-array fields for the specified string. Search for phrases, individual terms, and prefixes.

: search all and fields for the specified string. Search for phrases, individual terms, and prefixes. structured : search specific fields, construct compound queries using Boolean operators, and use advanced features such as term boosting and proximity searching.

: search specific fields, construct compound queries using Boolean operators, and use advanced features such as term boosting and proximity searching. lucene : specify search criteria using the Apache Lucene query parser syntax.

: specify search criteria using the Apache Lucene query parser syntax. dismax : specify search criteria using the simplified subset of the Apache Lucene query parser syntax defined by the DisMax query parser.

Parameters syntax:

$result = $client->search([

'cursor' => '<string>',

'expr' => '<string>',

'facet' => '<string>',

'filterQuery' => '<string>',

'highlight' => '<string>',

'partial' => true || false,

'query' => '<string>', // REQUIRED

'queryOptions' => '<string>',

'queryParser' => 'simple|structured|lucene|dismax',

'return' => '<string>',

'size' => <integer>,

'sort' => '<string>',

'start' => <integer>,

'stats' => '<string>',

]);

Since there are many of them, we will go through the most common in a search:

cursor <String> : Retrieves a cursor value you can use to page through large result sets. Use the size parameter to control the number of hits to include in each response. You can specify either the cursor or start parameter in a request; they are mutually exclusive. To get the first cursor, set the cursor value to initial .

: Retrieves a cursor value you can use to page through large result sets. Use the parameter to control the number of hits to include in each response. You can specify either the or parameter in a request; they are mutually exclusive. To get the first cursor, set the cursor value to . expr<String>: Defines one or more numeric expressions that can be used to sort results or specify search or filter criteria. You specify the expressions in JSON using the form {"EXPRESSIONNAME":"EXPRESSION"} . You can define and use multiple expressions in a search request. For example: {"expression1":"_score*rating", "expression2":"(1/rank)*year"}

Defines one or more numeric expressions that can be used to sort results or specify search or filter criteria. You specify the expressions in JSON using the form . You can define and use multiple expressions in a search request. For example: facet<String>: specifies one or more fields for which to get facet information, and options that control how the facet information is returned. You can specify the following faceting options: buckets, size, sort

specifies one or more fields for which to get facet information, and options that control how the facet information is returned. You can specify the following faceting options: query<String>: Specifies the search criteria for the request. How you specify the search criteria depends on the query parser used for the request and the parser options specified in the queryOptions parameter.

Specifies the search criteria for the request. How you specify the search criteria depends on the query parser used for the request and the parser options specified in the parameter. queryParser<String>: Specifies which query parser to use to process the request. If queryParser is not specified, Amazon CloudSearch uses the simple query parser. There are 4 types: simple : perform simple searches of text and text-array fields, structured : perform advanced searches by combining multiple expressions to define the search criteria, lucene : search using the Apache Lucene query parser syntax and dismax : search using the simplified subset of the Apache Lucene query parser syntax defined by the DisMax query parser

Specifies which query parser to use to process the request. If is not specified, Amazon CloudSearch uses the query parser. There are 4 types: : perform simple searches of and fields, : perform advanced searches by combining multiple expressions to define the search criteria, : search using the Apache Lucene query parser syntax and : search using the simplified subset of the Apache Lucene query parser syntax defined by the DisMax query parser size <long, int or float> : Specifies the maximum number of search hits to include in the response.

: Specifies the maximum number of search hits to include in the response. start<long, int or float>: Specifies the offset of the first search hit you want to return. Note that the result set is zero-based; the first result is at index 0. You can specify either the start or cursor parameter in a request, they are mutually exclusive.

This the common result syntax:

[

'facets' => [

'<String>' => [

'buckets' => [

[

'count' => <integer>,

'value' => '<string>',

],

// ...

],

],

// ...

],

'hits' => [

'cursor' => '<string>',

'found' => <integer>,

'hit' => [

[

'exprs' => ['<string>', ...],

'fields' => [

'<String>' => ['<string>', ...],

// ...

],

'highlights' => ['<string>', ...],

'id' => '<string>',

],

// ...

],

'start' => <integer>,

],

'stats' => [

'<String>' => [

'count' => <integer>,

'max' => '<string>',

'mean' => '<string>',

'min' => '<string>',

'missing' => <integer>,

'stddev' => <float>,

'sum' => <float>,

'sumOfSquares' => <float>,

],

// ...

],

'status' => [

'rid' => '<string>',

'timems' => <integer>,

],

]

2. Suggest:

Retrieves autocomplete suggestions for a partial query string. You can use suggestions enable you to display likely matches before users finish typing.

Parameters syntax:

$result = $client->suggest([

'query' => '<string>', // REQUIRED

'size' => <integer>,

'suggester' => '<string>', // REQUIRED

]);

query<String>: specifies the string for which you want to get suggestions.

specifies the string for which you want to get suggestions. size<long, int or float>: specifies the maximum number of suggestions to return.

specifies the maximum number of suggestions to return. suggester<String>: Specifies the name of the suggester to use to find suggested matches.

Result Syntax

[

'status' => [

'rid' => '<string>',

'timems' => <integer>,

],

'suggest' => [

'found' => <integer>,

'query' => '<string>',

'suggestions' => [

[

'id' => '<string>',

'score' => <integer>,

'suggestion' => '<string>',

],

// ...

],

],

]

3. UploadDocuments

Posts a batch of documents to a search domain for indexing. A document batch is a collection of add and delete operations that represent the documents you want to add, update, or delete from your domain. Batches can be described in either JSON or XML.

Parameter Syntax

$result = $client->uploadDocuments([

'contentType' => 'application/json|application/xml', // REQUIRED

'documents' => <string || resource || Psr\Http\Message\StreamInterface>, // REQUIRED

]);

contentType<String>: The format of the batch you are uploading. Amazon CloudSearch supports two document batch formats: application/json, application/xml

The format of the batch you are uploading. Amazon CloudSearch supports two document batch formats: application/json, application/xml documents<string|resource|Psr\Http\Message\StreamInterface>: A batch of documents formatted in JSON or HTML.

Result Syntax

[

'adds' => <integer>,

'deletes' => <integer>,

'status' => '<string>',

'warnings' => [

[

'message' => '<string>',

],

// ...

],

]

VII. Setting up Amazon CloudSearch

Setting up amazon CloudSearch is quite simple through the CloudSearch’s management console.

First we log in to CloudSearch using our credentials and create a domain

Tap on Create a new Domain

In the next screen choose your domain name and desired instance type:

We can have many options to configure our CloudSearch Domain, but in this tutorial we will use manual configuration

After that we will choose the field we want to create for CloudSearch

Next we will choose the access policy. Since this is a demo we will allow all access:

In the next screen, check to make sure everything configuration is correct then click confirm

After clicking confirm button it will take a while to index your CloudSearch.

When it’s done you can see the dashboard in the management tool

Starting from there you can upload your document to CloudSearch.

Resources