Extending Elasticsearch
Writing plugins, fast and easy!

Alexander Reelsen@spinscale
alex@elastic.co

Elasticsearch in 10 seconds
Search Engine (FTS, Analytics, Geo), real-timeDistributed, scalable, highly available, resilient
Interface: HTTP & JSON
Centrepiece of the Elastic Stack (Kibana, Logstash, Beats,APM, ML, App/Site/Enterprise Search)
Uneducated guess: Tens of thousands of clusters
worldwide, hundreds of thousands of instances

AgendaOverview
Building
TestingReleasing
Extension points

Overview
Requirements, gotchas and some details

Overview
Plugins are zip files
own jars/dependencies
loaded with its own classloader
own security permissions

Security Manager? Permissions? But why?!
Sandbox your java application
Prevent certain calls by your application
Policy file grants permissions
FilePermission (read, write)
SocketPermission (connect, listen, accept)
URLPermission, PropertyPermission, ...

Security Manager? Permissions? But why?!
Each plugin needs to be packaged with a policy thatgrants certain operations

Building
gradle clean assembleOverview
A plugin needs to be released for an Elasticsearch
patch release
Use the gradle plugin! Batteries included, lots of themIntegration tests, license checks, checkstyle, noticefiles

Testing
gradle clean check

Overview
Unit tests: ESTestCase
Integration tests: YAML
Download ES, install plugin, start ES, run tests
against HTTP

ESTestCase
Randomized testing
LoggerThread leak detector
Deprecation warnings detection
Test method names have to begin with void test...()
assertBusy/awaitBusy

Extension points
class MyPlugin extends Plugin

Overview
ActionPlugin: Implement own actions, REST endpoints
AnalysisPlugin: Add custom parts to analysis chain
ClusterPlugin: Implement own shard allocation logic
DiscoveryPlugin: Add (often API based) host discovery
IngestPlugin: Custom ingest processors
MapperPlugin: Custom mapping data types
NetworkPlugin: Network implementations (i.e. netty replacement)
RepositoryPlugin: Snapshot/restore repository implementations
ScriptPlugin: Script Engines, Script Contexts
SearchPlugin: Queries, Highlighter, Suggester, Aggregations, Rescoring, Search Extensions
ReloadablePlugin: Ensure a plugin can reload its stateOverview
main class: org.elasticsearch.plugins.Plugin
settings registration & filter
custom cluster state metadatacluster state listenerindex modules (search listener, index event listener)

Plugin Loading[INFO ][o.e.p.PluginsService] [node] loaded module [lang-mustache]Overview
PluginService
check for modules & plugins directorycheck jar hellcheck version
create classloader
reload lucene SPI
load Plugin classinstantiate Plugin class

Show me the code!

Ingest lang-detect processor
PUT _ingest/pipeline/langdetect-pipeline{"processors": [{"langdetect" : {"field" : "my_field","target_field" : "language"}}]}

Ingest lang-detect processor
PUT /my-index/my-type/1?pipeline=langdetect-pipeline{"my_field" : "This is hopefully an english text, that will bedetected."}
GET /my-index/my-type/1{"my_field" : "This is hopefully an english text, that will bedetected.","language": "en"}Custom query parser
Ecommerce use-caseclean data
non expert searches

Custom query parser

nike running hoodie xl

Custom query parser

nike running hoodie xl

Custom query parserbrand}}

nike running hoodie xlsizeCustom query parser
PUT my_products{"settings": {"index.queryparser.values" : {"brands" : [ "nike", "adidas", "puma", "salomon" ],"size" : [ "m", "xl", "l", "s", "xs", "xxs", "xxl" ]}}}Custom query parser
GET my_products/_search{"query": {"custom" : {"title": "nike running hoodie XL"}
}
}

DEMO

Summary
When is your plugin ready again?

Summary
Writing own plugins is not that hard
Testing plugins is incredibly easy with gradle
Release via github or sonatypeThanks for listening!
Questions?

Alexander Reelsen@spinscale
alex@elastic.coResources
https://github.com/elastic/elasticsearch/https://github.com/elastic/elasticsearch/tree/7.1/plugins/examples
https://github.com/spinscale/elasticsearch-ingest-opennlp
https://github.com/spinscale/elasticsearch-ingest-langdetecthttps://github.com/spinscale/cookiecutter-elasticsearch-ingest-processor

Thanks for listening!
Questions?

Alexander Reelsen@spinscale
alex@elastic.co