Magento 2.4 ElasticSerch vs. MySQL Search Performance
By: Yegor Shytikov
Tags: Elasticsearch, MySQL, Magento

With Magento 2, we could use an open-source solution such as Apache Solr or Elasticsearch. We could use a managed solution such as Klevu or Algolia or use MySQL full-text search, which was the default one before the M2.4 release. However, it is better to use MYSQL for the following reasons:

* There’s no need to install extra heavy Java software or libraries just to run a slow Magento 2.4.x website.

* You can use the same database interface we use in our app to perform search queries (PDO)

* You don’t need to provision any additional servers

* There’s no extra cost to you

* The data is stored in one place that you can easily control and debug

* There’s no requirement to keep your database rows in sync with another source

The main arguments against using MySQL search are accuracy and speed at scale. However, ElasticSearch with Magento 2 also has performance issues you will struggle with. It is not a silver bullet. So, the Magento sales team made a bad architectural decision to remove an easy solution vs. a more complex one.

So it is a solution to remove additional complexity and dependency and make your site faster:

https://www.yireo.com/blog/2020-09-26-removing-search-from-magento2

Elastic search profiling: Magento default store with sample data 3000 SKUs

Profiling:

We have the next elastic queries:

1 ) /magento2_product_1/document/_search

Started at:

214 ms

Duration:

2.42 ms (0.73%)

Memory Start:

16.8 MB

Memory Change:

6.7 KB (0.03%) Query: {

"from": 0,

"size": 12,

"stored_fields": [

"_id",

"_score"

],

"sort": [

{

"_score": {

"order": "desc"

}

}

],

"query": {

"bool": {

"must": [

{

"terms": {

"visibility": [

3,

4

]

}

}

],

"should": [

{

"match": {

"_search": {

"query": "shirt",

"boost": 2

}

}

},

{

"match": {

"description": {

"query": "shirt",

"boost": 2

}

}

},

{

"match": {

"manufacturer_value": {

"query": "shirt",

"boost": 2

}

}

},

{

"match": {

"name": {

"query": "shirt",

"boost": 6

}

}

},

{

"match": {

"short_description": {

"query": "shirt",

"boost": 2

}

}

},

{

"match": {

"sku": {

"query": "shirt",

"boost": 7

}

}

},

{

"match": {

"status_value": {

"query": "shirt",

"boost": 2

}

}

},

{

"match": {

"tax_class_id_value": {

"query": "shirt",

"boost": 2

}

}

},

{

"match": {

"url_key": {

"query": "shirt",

"boost": 2

}

}

},

{

"match": {

"_search": {

"query": "shirt",

"boost": 2

}

}

},

{

"match_phrase_prefix": {

"name": {

"query": "shirt",

"boost": 2

}

}

},

{

"match_phrase_prefix": {

"sku": {

"query": "shirt",

"boost": 2

}

}

}

],

"minimum_should_match": 1

}

},

"aggregations": {

"price_bucket": {

"extended_stats": {

"field": "price_0_1"

}

},

"category_bucket": {

"terms": {

"field": "categor...",

"size": 500

}

}

}

}

2) /magento2_product_1/document/_search

Started at:

217 ms Duration:

1.44 ms (0.43%) Query: {

"from": 0,

"size": 12,

"stored_fields": [

"_id",

"_score"

],

"sort": [

{

"_score": {

"order": "desc"

}

}

],

"query": {

"bool": {

"must": [

{

"terms": {

"visibility": [

3,

4

]

}

}

],

"should": [

{

"match": {

"_search": {

"query": "shirt",

"boost": 2

}

}

},

{

"match": {

"description": {

"query": "shirt",

"boost": 2

}

}

},

{

"match": {

"manufacturer_value": {

"query": "shirt",

"boost": 2

}

}

},

{

"match": {

"name": {

"query": "shirt",

"boost": 6

}

}

},

{

"match": {

"short_description": {

"query": "shirt",

"boost": 2

}

}

},

{

"match": {

"sku": {

"query": "shirt",

"boost": 7

}

}

},

{

"match": {

"status_value": {

"query": "shirt",

"boost": 2

}

}

},

{

"match": {

"tax_class_id_value": {

"query": "shirt",

"boost": 2

}

}

},

{

"match": {

"url_key": {

"query": "shirt",

"boost": 2

}

}

},

{

"match": {

"_search": {

"query": "shirt",

"boost": 2

}

}

},

{

"match_phrase_prefix": {

"name": {

"query": "shirt",

"boost": 2

}

}

},

{

"match_phrase_prefix": {

"sku": {

"query": "shirt",

"boost": 2

}

}

}

],

"minimum_should_match": 1

}

},

"aggregations": {

"prices": {

"extended_stats": {

"field": "price_0_1"

}

}

}

}

3) /magento2_product_1/document/_search

Started at:

218 ms

Duration:

1.31 ms (0.40%)

Query:

{

"from": 0,

"size": 12,

"stored_fields": [

"_id",

"_score"

],

"sort": [

{

"_score": {

"order": "desc"

}

}

],

"query": {

"bool": {

"must": [

{

"terms": {

"visibility": [

3,

4

]

}

}

],

"should": [

{

"match": {

"_search": {

"query": "shirt",

"boost": 2

}

}

},

{

"match": {

"description": {

"query": "shirt",

"boost": 2

}

}

},

{

"match": {

"manufacturer_value": {

"query": "shirt",

"boost": 2

}

}

},

{

"match": {

"name": {

"query": "shirt",

"boost": 6

}

}

},

{

"match": {

"short_description": {

"query": "shirt",

"boost": 2

}

}

},

{

"match": {

"sku": {

"query": "shirt",

"boost": 7

}

}

},

{

"match": {

"status_value": {

"query": "shirt",

"boost": 2

}

}

},

{

"match": {

"tax_class_id_value": {

"query": "shirt",

"boost": 2

}

}

},

{

"match": {

"url_key": {

"query": "shirt",

"boost": 2

}

}

},

{

"match": {

"_search": {

"query": "shirt",

"boost": 2

}

}

},

{

"match_phrase_prefix": {

"name": {

"query": "shirt",

"boost": 2

}

}

},

{

"match_phrase_prefix": {

"sku": {

"query": "shirt",

"boost": 2

}

}

}

],

"minimum_should_match": 1

}

},

"aggregations": {

"prices": {

"histogram": {

"field": "price_0_1",

"interval": 10,

"min_doc_count": 1

}

}

}

}

3) /magento2_product_1/document/_search

Started at:

220 ms Duration:

1.94 ms (0.58%) {

"from": 0,

"size": 12,

"stored_fields": [

"_id",

"_score"

],

"sort": [

{

"_score": {

"order": "desc"

}

}

],

"query": {

"bool": {

"must": [

{

"terms": {

"visibility": [

3,

4

]

}

}

],

"should": [

{

"match": {

"_search": {

"query": "shirt",

"boost": 2

}

}

},

{

"match": {

"description": {

"query": "shirt",

"boost": 2

}

}

},

{

"match": {

"manufacturer_value": {

"query": "shirt",

"boost": 2

}

}

},

{

"match": {

"name": {

"query": "shirt",

"boost": 6

}

}

},

{

"match": {

"short_description": {

"query": "shirt",

"boost": 2

}

}

},

{

"match": {

"sku": {

"query": "shirt",

"boost": 7

}

}

},

{

"match": {

"status_value": {

"query": "shirt",

"boost": 2

}

}

},

{

"match": {

"tax_class_id_value": {

"query": "shirt",

"boost": 2

}

}

},

{

"match": {

"url_key": {

"query": "shirt",

"boost": 2

}

}

},

{

"match": {

"_search": {

"query": "shirt",

"boost": 2

}

}

},

{

"match_phrase_prefix": {

"name": {

"query": "shirt",

"boost": 2

}

}

},

{

"match_phrase_prefix": {

"sku": {

"query": "shirt",

"boost": 2

}

}

}

],

"minimum_should_match": 1

}

},

"aggregations": {

"prices": {

"histogram": {

"field": "price_0_1",

"interval": 10,

"min_doc_count": 1

}

}

}

}

4) /magento2_product_1/product/_search

Started at

248 ms Duration

7.18 ms (2.2%) Network (outgoing)

8.3 KB Network (incoming)

3.8 KB Query {

"suggest": {

"text": "shirt",

"phrase_activity_value": {

"phrase": {

"field": "activit...",

"analyzer": "standard",

"size": 2,

"max_errors": 1,

"direct_generator": [

{

"field": "activit...",

"min_word_length": 3,

"min_doc_freq": 1

}

]

}

},

"phrase_category_gear_value": {

"phrase": {

"field": "categor...",

"analyzer": "standard",

"size": 2,

"max_errors": 1,

"direct_generator": [

{

"field": "categor...",

"min_word_length": 3,

"min_doc_freq": 1

}

]

}

},

"phrase_climate_value": {

"phrase": {

"field": "climate...",

"analyzer": "standard",

"size": 2,

"max_errors": 1,

"direct_generator": [

{

"field": "climate...",

"min_word_length": 3,

"min_doc_freq": 1

}

]

}

},

"phrase_collar_value": {

"phrase": {

"field": "collar_...",

"analyzer": "standard",

"size": 2,

"max_errors": 1,

"direct_generator": [

{

"field": "collar_...",

"min_word_length": 3,

"min_doc_freq": 1

}

]

}

},

"phrase_color_value": {

"phrase": {

"field": "color_v...",

"analyzer": "standard",

"size": 2,

"max_errors": 1,

"direct_generator": [

{

"field": "color_v...",

"min_word_length": 3,

"min_doc_freq": 1

}

]

}

},

"phrase_country_of_manufacture_value": {

"phrase": {

"field": "country...",

"analyzer": "standard",

"size": 2,

"max_errors": 1,

"direct_generator": [

{

"field": "country...",

"min_word_length": 3,

"min_doc_freq": 1

}

]

}

},

"phrase_custom_design_value": {

"phrase": {

"field": "custom_...",

"analyzer": "standard",

"size": 2,

"max_errors": 1,

"direct_generator": [

{

"field": "custom_...",

"min_word_length": 3,

"min_doc_freq": 1

}

]

}

},

"phrase_custom_layout_value": {

"phrase": {

"field": "custom_...",

"analyzer": "standard",

"size": 2,

"max_errors": 1,

"direct_generator": [

{

"field": "custom_...",

"min_word_length": 3,

"min_doc_freq": 1

}

]

}

},

"phrase_custom_layout_update_file_value": {

"phrase": {

"field": "custom_...",

"analyzer": "standard",

"size": 2,

"max_errors": 1,

"direct_generator": [

{

"field": "custom_...",

"min_word_length": 3,

"min_doc_freq": 1

}

]

}

},

"phrase_description": {

"phrase": {

"field": "descrip...",

"analyzer": "standard",

"size": 2,

"max_errors": 1,

"direct_generator": [

{

"field": "descrip...",

"min_word_length": 3,

"min_doc_freq": 1

}

]

}

},

"phrase_eco_collection_value": {

"phrase": {

"field": "eco_col...",

"analyzer": "standard",

"size": 2,

"max_errors": 1,

"direct_generator": [

{

"field": "eco_col...",

"min_word_length": 3,

"min_doc_freq": 1

}

]

}

},

"phrase_erin_recommends_value": {

"phrase": {

"field": "erin_re...",

"analyzer": "standard",

"size": 2,

"max_errors": 1,

"direct_generator": [

{

"field": "erin_re...",

"min_word_length": 3,

"min_doc_freq": 1

}

]

}

},

"phrase_features_bags_value": {

"phrase": {

"field": "feature...",

"analyzer": "standard",

"size": 2,

"max_errors": 1,

"direct_generator": [

{

"field": "feature...",

"min_word_length": 3,

"min_doc_freq": 1

}

]

}

},

"phrase_format_value": {

"phrase": {

"field": "format_...",

"analyzer": "standard",

"size": 2,

"max_errors": 1,

"direct_generator": [

{

"field": "format_...",

"min_word_length": 3,

"min_doc_freq": 1

}

]

}

},

"phrase_gender_value": {

"phrase": {

"field": "gender_...",

"analyzer": "standard",

"size": 2,

"max_errors": 1,

"direct_generator": [

{

"field": "gender_...",

"min_word_length": 3,

"min_doc_freq": 1

}

]

}

},

"phrase_gift_message_available_value": {

"phrase": {

"field": "gift_me...",

"analyzer": "standard",

"size": 2,

"max_errors": 1,

"direct_generator": [

{

"field": "gift_me...",

"min_word_length": 3,

"min_doc_freq": 1

}

]

}

},

"phrase_manufacturer_value": {

"phrase": {

"field": "manufac...",

"analyzer": "standard",

"size": 2,

"max_errors": 1,

"direct_generator": [

{

"field": "manufac...",

"min_word_length": 3,

"min_doc_freq": 1

}

]

}

},

"phrase_material_value": {

"phrase": {

"field": "materia...",

"analyzer": "standard",

"size": 2,

"max_errors": 1,

"direct_generator": [

{

"field": "materia...",

"min_word_length": 3,

"min_doc_freq": 1

}

]

}

},

"phrase_msrp_display_actual_price_type_value": {

"phrase": {

"field": "msrp_di...",

"analyzer": "standard",

"size": 2,

"max_errors": 1,

"direct_generator": [

{

"field": "msrp_di...",

"min_word_length": 3,

"min_doc_freq": 1

}

]

}

},

"phrase_name": {

"phrase": {

"field": "name",

"analyzer": "standard",

"size": 2,

"max_errors": 1,

"direct_generator": [

{

"field": "name",

"min_word_length": 3,

"min_doc_freq": 1

}

]

}

},

"phrase_new_value": {

"phrase": {

"field": "new_value",

"analyzer": "standard",

"size": 2,

"max_errors": 1,

"direct_generator": [

{

"field": "new_value",

"min_word_length": 3,

"min_doc_freq": 1

}

]

}

},

"phrase_options_container_value": {

"phrase": {

"field": "options...",

"analyzer": "standard",

"size": 2,

"max_errors": 1,

"direct_generator": [

{

"field": "options...",

"min_word_length": 3,

"min_doc_freq": 1

}

]

}

},

"phrase_page_layout_value": {

"phrase": {

"field": "page_la...",

"analyzer": "standard",

"size": 2,

"max_errors": 1,

"direct_generator": [

{

"field": "page_la...",

"min_word_length": 3,

"min_doc_freq": 1

}

]

}

},

"phrase_pattern_value": {

"phrase": {

"field": "pattern...",

"analyzer": "standard",

"size": 2,

"max_errors": 1,

"direct_generator": [

{

"field": "pattern...",

"min_word_length": 3,

"min_doc_freq": 1

}

]

}

},

"phrase_performance_fabric_value": {

"phrase": {

"field": "perform...",

"analyzer": "standard",

"size": 2,

"max_errors": 1,

"direct_generator": [

{

"field": "perform...",

"min_word_length": 3,

"min_doc_freq": 1

}

]

}

},

"phrase_price_view_value": {

"phrase": {

"field": "price_v...",

"analyzer": "standard",

"size": 2,

"max_errors": 1,

"direct_generator": [

{

"field": "price_v...",

"min_word_length": 3,

"min_doc_freq": 1

}

]

}

},

"phrase_quantity_and_stock_status_value": {

"phrase": {

"field": "quantit...",

"analyzer": "standard",

"size": 2,

"max_errors": 1,

"direct_generator": [

{

"field": "quantit...",

"min_word_length": 3,

"min_doc_freq": 1

}

]

}

},

"phrase_sale_value": {

"phrase": {

"field": "sale_va...",

"analyzer": "standard",

"size": 2,

"max_errors": 1,

"direct_generator": [

{

"field": "sale_va...",

"min_word_length": 3,

"min_doc_freq": 1

}

]

}

},

"phrase_shipment_type_value": {

"phrase": {

"field": "shipmen...",

"analyzer": "standard",

"size": 2,

"max_errors": 1,

"direct_generator": [

{

"field": "shipmen...",

"min_word_length": 3,

"min_doc_freq": 1

}

]

}

},

"phrase_short_description": {

"phrase": {

"field": "short_d...",

"analyzer": "standard",

"size": 2,

"max_errors": 1,

"direct_generator": [

{

"field": "short_d...",

"min_word_length": 3,

"min_doc_freq": 1

}

]

}

},

"phrase_size_value": {

"phrase": {

"field": "size_va...",

"analyzer": "standard",

"size": 2,

"max_errors": 1,

"direct_generator": [

{

"field": "size_va...",

"min_word_length": 3,

"min_doc_freq": 1

}

]

}

},

"phrase_sku": {

"phrase": {

"field": "sku",

"analyzer": "standard",

"size": 2,

"max_errors": 1,

"direct_generator": [

{

"field": "sku",

"min_word_length": 3,

"min_doc_freq": 1

}

]

}

},

"phrase_sleeve_value": {

"phrase": {

"field": "sleeve_...",

"analyzer": "standard",

"size": 2,

"max_errors": 1,

"direct_generator": [

{

"field": "sleeve_...",

"min_word_length": 3,

"min_doc_freq": 1

}

]

}

},

"phrase_status_value": {

"phrase": {

"field": "status_...",

"analyzer": "standard",

"size": 2,

"max_errors": 1,

"direct_generator": [

{

"field": "status_...",

"min_word_length": 3,

"min_doc_freq": 1

}

]

}

},

"phrase_strap_bags_value": {

"phrase": {

"field": "strap_b...",

"analyzer": "standard",

"size": 2,

"max_errors": 1,

"direct_generator": [

{

"field": "strap_b...",

"min_word_length": 3,

"min_doc_freq": 1

}

]

}

},

"phrase_style_bags_value": {

"phrase": {

"field": "style_b...",

"analyzer": "standard",

"size": 2,

"max_errors": 1,

"direct_generator": [

{

"field": "style_b...",

"min_word_length": 3,

"min_doc_freq": 1

}

]

}

},

"phrase_style_bottom_value": {

"phrase": {

"field": "style_b...",

"analyzer": "standard",

"size": 2,

"max_errors": 1,

"direct_generator": [

{

"field": "style_b...",

"min_word_length": 3,

"min_doc_freq": 1

}

]

}

},

"phrase_style_general_value": {

"phrase": {

"field": "style_g...",

"analyzer": "standard",

"size": 2,

"max_errors": 1,

"direct_generator": [

{

"field": "style_g...",

"min_word_length": 3,

"min_doc_freq": 1

}

]

}

},

"phrase_tax_class_id_value": {

"phrase": {

"field": "tax_cla...",

"analyzer": "standard",

"size": 2,

"max_errors": 1,

"direct_generator": [

{

"field": "tax_cla...",

"min_word_length": 3,

"min_doc_freq": 1

}

]

}

},

"phrase_url_key": {

"phrase": {

"field": "url_key",

"analyzer": "standard",

"size": 2,

"max_errors": 1,

"direct_generator": [

{

"field": "url_key",

"min_word_length": 3,

"min_doc_freq": 1

}

]

}

},

"phrase_visibility_value": {

"phrase": {

"field": "visibil...",

"analyzer": "standard",

"size": 2,

"max_errors": 1,

"direct_generator": [

{

"field": "visibil...",

"min_word_length": 3,

"min_doc_freq": 1

}

]

}

}

}

}

It is calgrap trace 5 calls Elasticsearch::query qula to number of our query.

Total Duration: 16.ms

It will be difficult to run all this query in legacy MySQL search lets start with the first one.

We will run this query against the 26K store.

The search table looks like this:

MariaDB [magento2]> select * from catalogsearch_fulltext_scope1 limit 1;

+-----------+--------------+----------------------------------------------+

| entity_id | attribute_id | data_index |

+-----------+--------------+----------------------------------------------+

| 3 | 71 | Sennheiser Gray Foam Mic Windscreen for ME66 |

+-----------+--------------+----------------------------------------------+

1 row in set (0.00 sec)

Search query:

340 rows in set (0.69 sec) MariaDB [magento2]> explain select * from catalogsearch_fulltext_scope1 where data_index like "%Windscreen%" ;

+------+-------------+-------------------------------+------+---------------+------+---------+------+--------+-------------+

| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |

+------+-------------+-------------------------------+------+---------------+------+---------+------+--------+-------------+

| 1 | SIMPLE | catalogsearch_fulltext_scope1 | ALL | NULL | NULL | NULL | NULL | 240391 | Using where |

+------+-------------+-------------------------------+------+---------------+------+---------+------+--------+-------------+

1 row in set (0.00 sec)

The query time is 0.69ms. Again Magento field doesn't have a full-text index :

ALTER TABLE catalogsearch_fulltext_scope1 ADD FULLTEXT(data_index)

And tuning this query again:

select SQL_NO_CACHE * from catalogsearch_fulltext_scope1 where MATCH(data_index) AGAINST(‘%Windscreen%’) ; 316 rows in set (0.01 sec)

Stop: where is the sow query?

Updated: I checked another Magento store that had Magento MYSQL search index and it has an index. Magento version is 2.3.3:

catalogsearch_fulltext_scope1 | 1 | FTI_FULLTEXT_DATA_INDEX | 1 | data_index | NULL | NULL | NULL | NULL | YES | FULLTEXT

So, With index MySQL Serch is fast you need just to know how to search.

The same Elastic Search Query:

time curl -X GET "localhost:9200/magento2_product_1/document/_search?request_cache=false" -H 'Content-Type: application/json' -d'

{

"query": {

"match": {

"_search": {

"query": "shirt"

}

}

}

}

'

{"took":1,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":{"value":5,"relation":"eq"},"max_score":3.9976969,"hits":[..] real 0m0.008s

user 0m0.006s

sys 0m0.000s

Query took: 0.008s ~ 0.01s

Yes, Elastic Search is a more advanced solution that has aggregations.

Elasticsearch organizes aggregations into three categories:

Metric aggregations that calculate metrics, such as a sum or average, from field values.

aggregations that calculate metrics, such as a sum or average, from field values. Bucket aggregations that group documents into buckets, also called bins, based on field values, ranges, or other criteria.

aggregations that group documents into buckets, also called bins, based on field values, ranges, or other criteria. Pipeline aggregations that take input from other aggregations instead of documents or fields.

However, it is now faster for the same simple use cases.

The main difference ElasticSearch from MySQL-search is that ES works faster when large amounts of data through indexing. The index contains ready-made sets of data with which you are operating further ES-filters. So if you search with ES, you haven’t to make multiple aggregation N+1 direct requests to the database, as in MySQL.However, you can easily achieve the same functionality with Node.JS async I/) nonblocking microservices. This is the main reason to use ElasticSearch in HighLoad projects. For a small project, you will not feel the difference. Autocomplete works god with MySQL search also jut run explain query. The biggest issue Magento autocomplete 3-d party extensions are purely written. By the way, Magento doesn’t provide auto-complete functionality at all.

And what we see also with elastic search Magento queries the same famous Magento innovation EAV tables. Stop, how about the flat table in the Elastic Search? Magento just broke, and purely written software and nobody actually knows how it works. If it at least work don’t fix! So, it doesn’t really matter what your store will use for search Mysql or ES, it will use the same broken M2 core and will have the same bad performance as all Magento 2 websites have, and only Varnish will save you. However, how will you pre-cache and cache warm user-defined random search queries? Pure merchants and their customers…