Introduction into Elasticsearch& Spring Data ElasticsearchAlexander Reelsen
Community Advocate
alex@elastic.co | @spinscale

TOC
Why do you need a search engine in your app?
Introduction into Elasticsearch
Introduction into Spring Data Elasticsearch
DemoRunning Elasticsearch: Scaling your cluster
Next steps

Why do you need a search engine?
... or any data storeSpeed, Scale & RelevanceSpeedScaleRelevance... and much more
NRT: Searching & Indexing
Read scalability & write scalability
Resiliency
Operational simplicity & monitoring capabilities
Developer experience
Infrastructure integration
Team experience
Use-Cases: Observability, Workplace Search, Security, Product Search, Wikipedia

Product Overview

Solutions

Elastic Stackbuilding blocks

Deployment options

Licensing

Elastic Stackbuilding blocks

Elasticsearch in 10 seconds
Search Engine (FTS, Analytics, Geo), near real-timeDistributed, scalable, highly available, resilient
Interface: HTTP & JSON
Heart of the Elastic Stack (Kibana, Logstash, Beats)

Installation & Start# https://www.elastic.co/downloads/elasticsearch
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.9.1-darwin-x86_64.tar.gz# wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.9.1-linux-x86_64.tar.gz# wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.9.1-windows-x86_64.ziptar zxf elasticsearch-7.9.1-darwin-x86_64.tar.gz
cd elasticsearch-7.9.1./bin/elasticsearch

wget https://artifacts.elastic.co/downloads/kibana/kibana-7.9.1-darwin-x86_64.tar.gz# wget https://artifacts.elastic.co/downloads/kibana/kibana-7.9.1-linux-x86_64.tar.gz# wget https://artifacts.elastic.co/downloads/kibana/kibana-7.9.1-windows-x86_64.ziptar zxf kibana-7.9.1-darwin-x86_64.tar.gz
cd kibana-7.9.1./bin/kibana

Point your browser to http://localhost:5601/

Click Dev-Tools
Samples in Kibana
Samples in GithubIntroduction into Spring Data ElasticsearchCommunity maintained Spring Data Extension
Reactive extension
Make sure to use major version 4 (based on Elasticsearch 7.x), default in Spring
Boot 2.3
Uses the Elasticsearch REST Client

Elasticsearch REST client
Depends on the Elasticsearch core project
Based on Apache HTTP Client (works on java 8), might want to consider shading
Supports synchronous calls & cancellable async calls
Threadsafe
RestClient
RestHighLevelClient

Elasticsearch REST client architectureSpring Data Elasticsearch - Basics
ElasticsearchTemplate & ElasticsearchRestTemplate
MappingElasticsearchConverter
CrudRepository

Auditing, Entity Callbacks, eﬃcient scroll searchingSpring Data Elasticsearch - Entities@Document(indexName = "persons", shards = 1, createIndex = false)
public class Person {@Idprivate String id;
private String name;@Email@Field(type = FieldType.Keyword)
private String email;@Field(name="created_at", type = FieldType.Date, format = DateFormat.date_time)
private Date createdAt;@Size(max=500)@Pattern(regexp = "https?://.*", message = "must start with http:// or https://")@URL@Field(type = FieldType.Keyword)
private String url;}

private List<Person> friends; // creates an arrayprivate Point Location; // maps to geo_point

Spring Data Elasticsearch - Repositories
import org.springframework.data.elasticsearch.repository.ElasticsearchRepository;
public interface UserProfileRepository extends ElasticsearchRepository<UserProfile, String> {}Dynamic ﬁnders like findByEmail(String email)Attention: Ineﬃcient queries like findByDescriptionEndingWith()Spring Data Elasticsearch - Searching
final BoolQueryBuilder qb = QueryBuilders.boolQuery()// somewhat stable randomization to make sure users get an arbitrary document.must(QueryBuilders.scriptScoreQuery(QueryBuilders.matchAllQuery(), new Script("randomScore(1000, 'created_at')")))// only consider contributions that are not yet approvaed.filter(QueryBuilders.termQuery("state", Contribution.State.CREATED.name()))// ensure only contributions from the same region are shown.filter(QueryBuilders.termQuery("region", profile.getRegion()))// only consider languages spoken by the user as well as english.filter(QueryBuilders.termsQuery("language", languages))// exclude documents that were created by this user.mustNot(QueryBuilders.termQuery("submitted_by.email", profile.getEmail()))// exclude documents that were already voted on.mustNot(QueryBuilders.termQuery("comments.submitted_by.email", profile.getEmail()));
Query query = new NativeSearchQuery(qb).setPageable(PageRequest.of(0, 1));
final SearchHit<Contribution> result = elasticsearchRestTemplate.searchOne(query, Contribution.class);

Spring Data Elasticsearch - Countprivate boolean canSubmitMoreContributions(String email) {final BoolQueryBuilder qb = QueryBuilders.boolQuery().filter(QueryBuilders.termQuery("submitted_by.email", email)).filter(QueryBuilders.rangeQuery("created_at").gte("now-1d"));final long recentlySubmittedCount = elasticsearchRestTemplate.count(new NativeSearchQuery(qb), Contribution.class);
return recentlySubmittedCount <= 10;}

Spring Data Elasticsearch - Countpublic interface ContributionRepository extends ElasticsearchRepository<UserProfile, String> {@Query("{\"bool\": { \"must\" : [ { \"term\" : { \"submitted_by.email\":\"?0\" } }, { \"range\" : { \"created_at\": { \"gte\" : \"?1\" } } } ] } }")long countRecentContributions(String email, String date);}Spring Data Elasticsearch - Aggregations// filter by approved
final BoolQueryBuilder qb = QueryBuilders.boolQuery().filter(QueryBuilders.termQuery("state", Contribution.State.APPROVED.name())).filter(QueryBuilders.termQuery("region", region.name()));
final NativeSearchQuery query = new NativeSearchQuery(qb);// aggregate on username, get top 10, sum up score
query.addAggregation(AggregationBuilders.terms("by_user").field("submitted_by.email").size(40).subAggregation(AggregationBuilders.sum("total_score").field("score"))// make sure we get the full name of the last contribution.subAggregation(AggregationBuilders.topHits("by_name").size(1).sort(SortBuilders.fieldSort("created_at").order(SortOrder.DESC)).fetchSource("submitted_by.full_name", "")));
query.setPageable(Pageable.unpaged());
final SearchHits<Contribution> hits = elasticsearchRestTemplate.search(query, Contribution.class);// returns an Elasticsearch class
final Aggregations aggregations = hits.getAggregations();DemoRunning Elasticsearch: Scaling your cluster
Do not overshard: Single shard can easily contain 20-50GB
Let the ﬁlesystem cache get to work
Performance test, on your data! Use rally
Hint: Capacity Planning Webinar

Compute Resources
Storage: SSDs for hot data, HDDs for warm/cold, avoid NAS
Memory: JVM heap + OS cacheCompute: Thread pool scaling based on CPU countNetwork: The faster the better (careful cloud providers with burst rates)

Next steps
Improve your search: Learn about mappings and queries
Improve your model
Figure out expected throughput
Use aliases, always!

Summary
Search is never done!
Use the reference documentation
Ask your users about expectations, do not guess!
Testing: TestContainers

Resources
spinscale/link-rating
QoverySpring Data Elasticsearch Documentation
Elasticsearch Java REST Client Documentation
Elasticsearch Nightly Benchmarks

Thanks for listening
Q&A
Alexander Reelsen
Community Advocate
alex@elastic.co | @spinscaleElastic CloudElastic Support
Subscriptions

Discuss Forumhttps://discuss.elastic.coCommunity & Meetups
https://community.elastic.co

Elastic YouTube
Community
https://ela.st/yt-community

Thanks for listening
Q&A
Alexander Reelsen
Community Advocate
alex@elastic.co | @spinscale