Hortonworks Data Platform 2.6.5 In-Place Upgrade to Cloudera Data Platform 7
Tags: Cloudera, Cdp, Hdp, Hortonworks, Ambari

Hortonworks Data Platform 2.6.5 In-Place Upgrade to Cloudera Data Platform 7 Hashmap Follow Sep 10 · 8 min read

by Laurent Weichberger and Shubham Nikam

Since Cloudera and Hortonworks joined forces last year, customers have been operating on legacy versions of both CDH and HDP. For anyone still operating on HDP 2.6.5, there is a short window until Cloudera moves that version into sunset mode (end of life) on December 31, 2020.

Any major upgrade should go through significant planning and decision-making prior to kicking it off. In this blog post, we will cover our experience in achieving an “in-place” upgrade of HDP 2.6.5 to the newly released CDP 7. We hope it helps you in determining your path forward and making the various decisions along the way.

Background for the Upgrade

In order to have the hands-on experience of upgrading to CDP 7, I decided to perform an in-place upgrade of HDP 2.6.5 myself. I knew that there were many steps and stages to this work, and I had been working with my extended Hashmap team and David Fowler in the Cloudera Partner Enablement Team at Cloudera to create a definitive upgrade guide. Dave was a fantastic resource, who provided me with tons of documentation on how to accomplish this.

The challenging part was actually doing each and every step in the right order. Since our Hashmap team had created a guide of our own, I decided to walk our own talk. Also, this would allow me to enrich the guide as I went through each point myself. I hope you find this useful in your own work. Here is what I discovered.

Let’s get started

Before the upgrade can be attempted, Ambari itself must be updated.

In order for this Ambari upgrade to work, we have to disable the “Auto-Start of Services.” Hortonworks had created a cluster diagnostic tool named SmartSense, and this as well as Ambari Metrics must be put into “Maintenance Mode.”

Ambari Metrics alone has a pre-upgrade Linux shell script that needs to be downloaded from “archive.cloudera.com” and run on the server.

After this script, the entire Ambari Metrics service itself is stopped.

3. Then we need to stop the Ambari Server itself and all the Ambari Agents, and backup the Ambari database.

4. The Ambari Server has to be upgraded to an interim version before it can be transformed into Cloudera Manager.

5. Next, we need to download the new Ambari repository, with the intermediate code in it.

This is basically a secured Wget command asking that same Cloudera archive to download “ ambaridc.repo. ”

” When I spoke about details, we actually manually edit the repo to modify the content: vi /etc/yum.repos.d/ambaridc.repo

Then we need to manually visually validate that the repo has the correct Ambari version available for upgrade with yum: yum info ambari-server

Once we are satisfied with the version we can upgrade Ambari: yum upgrade ambari-server

6. Then we have to upgrade Ambari Agents on all the hosts: yum upgrade ambari-agent

7. We also need to upgrade the Ambari Server database itself: ambari-server upgrade

8. Then we start the Ambari Server again, and this time it is an interim version that is not supported except in the upgrade process.

We are now neither here nor there, we are not on HDP really, and not on CDP, but in the middle and everything is lit up with red Christmas bulbs, making it feel a bit alarming.

9. If you have configured Ambari to authenticate against an external LDAP or Active Directory, you must re-run: ambari-server setup-ldap

10. If you have configured your cluster for Hive, Ranger or Oozie with an external database, you have to make sure Ambari Server knows how to connect using the correct Driver: ambari-server setup — jdbc-db=mysql — jdbc-driver=/usr/share/java/mysql-connector-java.jar

We are basically at the halfway point and can rest for a moment. Whew!

Before we can complete the upgrade there are plenty of more steps including:

11. Security considerations, such as:

Encrypt Database and LDAP Passwords in Ambari.

Provide a master key for encrypting the passwords.

Kerberos configurations.

12. Next, we perform pre-upgrade checks:

Do we have enough room on the file-systems on the cluster servers to perform the in-place upgrade?

Are all Ambari agents heart-beating to the Ambari server?

The Hive Metastore needs to be backed up. Do any Hive tables require compaction?

Nothing can be in “Maintenance Mode” during the upgrade, since this is in-place (except for Ambari Metrics System, SmartSense, and Log Search).

and Make sure all the services are restarted and all service checks must pass.

Different services have their own databases, and these need to be backed up as well.

If NameNode is in HA, HDFS needs a checkpoint.

13. Next, we work with the Stack and Versions in HDP 2.6.5, so that we can arrive at the ability to upgrade. At this point, there is no upgrade button to push.

We can clearly see here where we came from. But where are we going?

14. In order to proceed with this upgrade, we use the “MANAGE VERSIONS” function.

15. Then we “REGISTER VERSION” and provide an XML file to give Ambari the information it needs to rebuild itself: HDP-7.1.1.0–565.xml

Then we use “Install Packages” and we are getting closer. Still no upgrade button though.

This is a required interim step. Soon we will have the path to upgrade.

16. Once the “packages” are installed, there is a surprise for us:

However, it is still premature to push the “PERFORM UPGRADE.”

There is still more preparation to perform prior to the upgrade, as we shall see, but we have made progress!

One thing that was a minor roadblock on my upgrade project here was the installed version of Java was wrong.CDP requires a specific version of Java or newer and so the upgrade steps were failing because of this. We fixed this problem with: yum install java-1.8.0-openjdk-devel

Furthermore, we needed to make sure .bashrc file has the right details, or else the upgrade process gets lost in space:

export JAVA_HOME=/usr/jdk64/jdk<version>

export PATH=$JAVA_HOME/bin:$PATH

17. Hive is a major consideration. If you are using Hive ACID tables, there is some pre-work here. Ambari configuration properties need to be carefully set to enable ACID:

hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager

hive.support.concurrency=true

18. We want to make Hive tables compatible with SparkSQL so we need to externalize them: ALTER TABLE <table name> SET TBLPROPERTIES(‘EXTERNAL’=’TRUE’,’external.table.purge’=’true’);

19. There is a Hive pre-upgrade tool JAR, for compaction, which we download from “repo.hortonworks.com” and subsequently run the downloaded Java utility: java PreUpgradeTool

20. You must dump the Hive metastore metadata immediately before upgrading.

21. We also upgrade Ambari Metrics system and SmartSense.

22. There are some optional steps we will not discuss here because I want to get you to the point where you can push that [UPGRADE] button.

Click [UPGRADE] on the target version: “HDP-7.1.1.0”

on the target version: “HDP-7.1.1.0” There are two types of upgrades, Rolling and Express. We are going to do Express because currently, this is the only supported way to get from HDP 2.6.5 to CDP 7.

Click [PROCEED] and it will ask you to confirm this decision.

You will see all sorts of work being done for you as the window of finished work expands, and a progress bar grows.

When it is done, it will show you the results:

Your “Current” version will now be HDP 7.1.x.y

23. Next, there is a lot of updating required now for Ranger, Hive migration, Ambari Metrics, and other things.

24. And then update the Spark SQL warehouse location.

25. Then we prepare to transition from this HDP 7 to the Cloudera Data Platform itself.

Back up Spark.

Back up Cluster Topology.

Making sure HDFS is healthy, etc.

26. Finally, we need to install and configure Cloudera Manager, as Ambri will go away, and never come back.

From the Cloudera archive we used previously, we download: cloudera-manager.repo

We actually manually edit this repo, believe or not.

We install all this goodness: yum install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server

27. Then we do quite a bit of work to configure the MySQL Database, without which we cannot be successful. It needs to look like this before we can continue:

28. Then we prepare a Cloudera Manager database, using a Linux shell script provided: scm_prepare_database.sh

29. Shortly after that, run the Cloudera Manager: service cloudera-scm-server start

Oh, and I saved the best for last. You will see this landing page and know that you are successful:

Where Do You Go From Here?

That should be more than enough for you to get a sense of what this upgrade entails. It is obviously a multi-step process. Also, keep in mind that if you are using Hortonworks Data Flow (HDF), the HDF upgrade to CDF is a different upgrade process altogether.

If you’d like assistance along the way, then please contact us. We’ve been working with Cloudera and Hortonworks since 2012 and would be glad to partner with you on anything from assessment and strategy to upgrade implementation.

Hashmap offers a range of enablement workshops and assessment services, cloud modernization and migration services, and consulting service packages as part of our cloud migration and modernization service offerings.

Listen to “Hashmap on Tap” and Watch “Megabytes”