Project: Beam (App Definition and Development / Streaming & Messaging)
File: hop.md

title: "Beam visual pipeline development with Hop" name: "Neo4j" icon: /images/logos/powered-by/hop.png category: study cardTitle: "Visual Apache Beam Pipeline Design and Orchestration with Apache Hop" cardDescription: "Apache Hop is an open source data orchestration and engineering platform that extends Apache Beam with visual pipeline lifecycle management. Neo4js Chief Solution Architect and Apache Hops co-founder, Matt Casters, sees Apache Beam as a driving force behind Hop." authorName: "Matt Casters" authorPosition: "Chief Solutions Architect, Neo4j, Apache Hop co-founder" authorImg: /images/mattcastersphoto.png publishDate: 2022-02-15T12:21:00+00:00 <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> <div class="case-study-opinion"> <div class="case-study-opinion-img"> <img class="case-study-opinion-img-center" src="/images/logos/powered-by/hop.png"/> </div> <blockquote class="case-study-quote-block"> <p class="case-study-quote-text"> Apache Beam and its abstraction of the execution engines is a big thing for us. The amount of work that that saves...it would be hard to build that support for Dataflow or Spark all by yourself. It is amazing that this technology exists in the first place, really amazing! Not having to worry about all those underlying platforms - that is tremendous! </p> <div class="case-study-quote-author"> <div class="case-study-quote-author-img"> <img src="/images/mattcastersphoto.png"> </div> <div class="case-study-quote-author-info"> <div class="case-study-quote-author-name"> Matt Casters </div> <div class="case-study-quote-author-position"> Chief Solutions Architect, Neo4j, Apache Hop co-founder </div> </div> </div> </blockquote> </div> <div class="case-study-post"> is an open source data orchestration and data engineering platform that aims to facilitate all aspects of data processing with visual pipeline development environment. This easy-to-use, fast, and flexible platform enables developers to create and manage Apache Beam batch and streaming pipelines in Hop GUI. Apache Hop uses metadata and kernel to describe how the data should be processed, and Apache Beam to design once, run anywhere. Chief Solutions Architect, , has been an early adopter of Apache Beam and its abstraction of execution engines. Matt has been an active member of the Apache open-source community for years and has leveraged Apache Beam as an execution engine to build Apache Hop. Thriving popularity and the growing number of Apache Beam users across the globe inspired Matt Casters to expand the idea of abstraction to visual pipeline lifecycle management and development. Matt co-founded and incubated the project that became a top level project at the in December 2021. The platform enables users of all skill levels to build, test, launch, and deploy powerful data workflows without writing code. Apache Hops intuitive drag and drop interface provides a visual representation of Apache Beam pipelines, simplifying pipeline design, execution, preview, monitoring, and debugging. <blockquote class="case-study-quote-block case-study-quote-wrapped"> <p class="case-study-quote-text"> I was a big fan of Beam from the get go. Apache Beam is now a very important part of the Apache Hop project. </p> <div class="case-study-quote-author"> <div class="case-study-quote-author-img"> <img src="/images/mattcastersphoto.png"> </div> <div class="case-study-quote-author-info"> <div class="case-study-quote-author-name"> Matt Casters </div> <div class="case-study-quote-author-position"> Chief Solutions Architect, Neo4j, <br>Apache Hop co-founder </div> </div> </div> </blockquote> The Apache Hop GUI allows data professionals to work visually and focus on what they need to do rather than how, using metadata to describe how the Apache Beam pipelines should be

Apache Hops action plugins () link transforms together, creating a pipeline. Various Apache Beam runners, such as , , , and the runner, read the metadata with help of Apache Hop's and , and execute the pipeline. Apache Hops custom plugins and metadata objects for , such as , empower users to execute database- and technology-specific transforms inside the Apache Beam pipelines, which allows for native optimized connectivity and flexible Apache Beam pipeline configurations. For instance, the Apache Hops stores logging and execution lineage of Apache Beam pipelines in the Neo4j graph database and enables users to query this information for more details, such as quickly jump to the place where an error occurred. The combination of Apache Hop transforms, , and Apache Beam-powered data processing opens up new horizons for more sinks and sources and custom use cases. Apache Hop aims to bring a no-code approach to Apache Beam data pipelines. Sometimes the choice of a particular programming language, framework, or engine is driven by developers' preferences, which results in businesses becoming tied to a specific technology skill set and stack. Apache Hop eliminates this dependency by abstracting out the I/Os with a fully pluggable runtime support and providing a graphic user interface on top of Apache Beam pipelines. All settings for pipeline elements are performed in the Hops visual editor just once, and pipeline is automatically described as metadata in JSON and CSV formats. Programming data pipelines source code becomes an option, not a necessity. Apache Hop does not require knowledge of a particular programming language to create pipelines, helping with the adoption of Apache Beam unified streaming and batch processing technology. <blockquote class="case-study-quote-block case-study-quote-wrapped"> <p class="case-study-quote-text"> In general, a visual pipeline design interface is really valuable for a non-developer audience We categorically choose the side of the organization when it comes to lowering setup costs, maintenance costs, increasing ROI, and safeguarding an investment over time. </p> <div class="case-study-quote-author"> <div class="case-study-quote-author-img"> <img src="/images/mattcastersphoto.png"> </div> <div class="case-study-quote-author-info"> <div class="case-study-quote-author-name"> Matt Casters </div> <div class="case-study-quote-author-position"> Chief Solutions Architect, Neo4j, <br>Apache Hop co-founder </div> </div> </div> </blockquote> Apache Beam continuously expands the number of use cases and scenarios it supports and makes it possible to bring advanced technology solutions into a reality. Being an early adopter of Apache Beam and its powerful abstraction, Matt Casters leveraged this knowledge and experience to create Apache Hop. The platform creates a value-add for Apache Beam users by enabling visual pipeline development and lifecycle management. Matt sees Apache Beam as a foundation and a driving force behind Apache Hop. Communication between Apache Beam and Apache Hop projects keeps fostering co-creation and enriches both products with new features. Apache Hop project is the example of the continuous improvement driven by the Apache open source community and amplified by collaborative organizations. <blockquote class="case-study-quote-block case-study-quote-wrapped"> <p class="case-study-quote-text"> Knowledge sharing and collaboration is something that comes naturally in the community. If we see some room for improvement, we exchange ideas and this way, we keep driving Apache Beam and Apache Hop projects forward. Together, we can work with the most complex problems and just solve them. </p> <div class="case-study-quote-author"> <div class="case-study-quote-author-img"> <img src="/images/mattcastersphoto.png"> </div> <div class="case-study-quote-author-info"> <div class="case-study-quote-author-name"> Matt Casters </div> <div class="case-study-quote-author-position"> Chief Solutions Architect, Neo4j, <br>Apache Hop co-founder </div> </div> </div> </blockquote> {{< casestudyfeedback Hop >}} </div> <div class="clear-nav"></div>