Ethtool Support for Microsoft Hyper-V

Ethtool Support for Microsoft Hyper-V is a programming project to implement support of the Linux command-line application "ethtool" on virtual machine guests of the Microsoft Hyper-V hypervisor. Ethtool is used to query and control network driver and hardware settings. Enabling support requires patching the virtual network driver in the Linux kernel.
===Problem Statement===
Ethtoolhttps://www.kernel.org/pub/software/network/ethtool/ is a command-line utility for Linux-kernel based operated systems, maintained in parallel with the kernel. Ethtool is primarily used by system administrators to query and/or set network driver variables such as link status, number of send and receive queues of vRSS, size of send and receive buffers, available features, and gathered statistics.
The network driver used by Linux virtual machines running on Microsoft Hyper-Vhttps://en.wikipedia.org/wiki/Hyper-V minimally supports ethtool; it can return the name of the driver, "hv_netvsc," and the permanent address of the network device. All Linux-supported network drivers are responsible for implementing the ethtool API. The goal of this project is to implement support for all common uses of ethtool.
====Background====
A common scenario for system administrators is to host multiple virtual machineshttps://en.wikipedia.org/wiki/Virtual_machine on the same piece of hardware, so one physical computer and its host operating system can support many guest computers, each with its own operating system. This process of "virtualization" https://en.wikipedia.org/wiki/Virtualization is dependent on an abstraction layer called a hypervisorhttps://en.wikipedia.org/wiki/Hypervisor. Microsoft's solution is a native hypervisor, Microsoft Hyper-V, included in most versions of Microsoft Windows. A native hypervisor runs directly on the host's hardware, in contrast to a hosted hypervisor which runs on the host's operating system, so the former has fewer layers of translation for guest operating system instructions.
====Specifications====
Since the guest operating systems are separate from the host operating system, a host machine running Windows with Hyper-V supports guestshttp://azure.microsoft.com/en-us/documentation/articles/virtual-machines-linux-opensource/ running Linux (as well as Windows). Linux supports being run as a guest operating system on a hypervisor through virtual drivers for the guest's virtual hardware, including video, storage, keyboard, mouse and networking. These drivers are a part of the Linux kernel, and are maintained by our client, the Microsoft Linux Integration Services team. We are concerned with the virtual network driver, netvschttps://git.kernel.org/cgit/linux/kernel/git/davem/net-next.git/tree/drivers/net/hyperv.
===Design===
Ethtool is a mature application and its programminghttps://git.kernel.org/cgit/linux/kernel/git/davem/net-next.git/tree/include/linux/ethtool.h interfacehttps://git.kernel.org/cgit/linux/kernel/git/davem/net-next.git/tree/include/uapi/linux/ethtool.h in the kernel is well-established. In general, adding support for each ethtool operation requires patching netvschttps://git.kernel.org/cgit/linux/kernel/git/davem/net-next.git/tree/drivers/net/hyperv/netvsc_drv.c with the appropriate new callback for the operation and any additional necessary changes.
====Goals====
Our current goal is to implement support for the following ethtool operations:
Table 1: Implementation Goals	Table 1: Implementation Goals 	Table 1: Implementation Goals
Operation 	Description 	Status
show-channels 	Queries the network device for the numbers of channels it has. A channel is an IRQ and the set of queues that can trigger that IRQ.	Complete
set-channels 	Changes the numbers of channels of the specified network device. 	Testing
show-features 	Queries the specified network device for the state of protocol offload and other features. 	Stretch Goal
show-ring 	Queries the specified network device for rx/tx ring parameter information. 	Stretch Goal
set-ring 	Changes the rx/tx ring parameters of the specified network device. 	Stretch Goal
statistics 	Queries the specified network device for network interface card and driver specific statistics. 	Stretch Goal
====Implementation====
Our work in progress is tracked on our issues pagehttps://suchcodemuchlove.com/academic/net-next/issues.
Currently, we have implemented the show-channelshttps://lkml.org/lkml/2015/2/26/727 operation. Our patch was signed-offhttps://lkml.org/lkml/2015/2/27/342 by Haiyang Zhang, and appliedhttps://lkml.org/lkml/2015/2/28/244 to the net-next kernel tree by Dave Miller. The patch was commitedhttps://git.kernel.org/cgit/linux/kernel/git/davem/net-next.git/commit/?id=59995370dbca7636c105ddadc0447fab86ad3887 and is now in the development version of the Linux kernel.
Our next goal is to implement the set-channels operation. We several issues to finalize before our final patch submission. These issues include:
Table 2: Current Issues 	Table 2: Current Issues 	Table 2: Current Issues	Table 2: Current Issues
Major Requirements 	Description 	Operation 	Status
work syncinghttps://suchcodemuchlove.com/academic/net-next/issues/4 	Determine the proper way to pause and re-sync the work over the new channels.	set-channel 	Closed
NVSP versionhttps://suchcodemuchlove.com/academic/net-next/issues/3 	Determine if every NVSP version can support changing num_chn. 	set-channel 	Closed
netif_set_real_num_tx_queues()https://suchcodemuchlove.com/academic/net-next/issues/2	Develop error checking since netif_set_real_num_tx_queues() can fail. 	set-channel 	Closed
Host-side Settingshttps://suchcodemuchlove.com/academic/net-next/issues/1 	Check if host supports RSS, Intel x520 etc. Ensure VMQ is enabled. 	set-channel 	Closed
====Testing====
Testing patches to a virtual network driver on a Linux machine is a tedious process. It requires direct access to a host machine running Windows with Hyper-V, and a Linux virtual machine for development and testing. When testing the network driver your machine is dependent on, if your patch causes the driver to break, direct access through the console is needed to debug, as network access will be disabled.
Additionally, developing Linux kernel patches requires configuring and installing the correct development version of the Linux kernel to the test machine. For networking drivers, we work off Dave Miller's net-next treehttps://git.kernel.org/cgit/linux/kernel/git/davem/net-next.git/. Our team primarily develops on CentOS 7 virtual machines, since installing a locally compiled kernel is made relatively easy with the kernel build system's creation of a native package.
We wrote a Linux development compilation reference guidehttps://suchcodemuchlove.com/academic/net-next/wikis/home kernel for each time test machine we setup.
Our normal workflow is to develop on a local clone of the kernel tree, and then push the patch to our test machine, where we can compile and install the patched netvsc module. When our patch breaks, we use the Remote Desktop Protocol to connect to the host Windows machine, to then use the Hyper-V manager's to connect to the Linux virtual machine without requiring the guest's networking to be operational. Further debugging is done through liberal use of printk, the kernel's version of printf for printing out debug information.
Testing the patches under load requires an extensive hardware setup: a Windows Server 2012 R2 machine running Hyper-V, with a 1 Gbps network adapter which supports Receive Side Scaling, so that the virtual machines can use virtual RSS. This resource is currently being provisioned.
====Project Deliverables====
===== get-channels =====
* From: Andrew Schwartzmeyer <andrew@schwartzmeyer.com>
* Date: Wed, 25 Feb 2015 19:49:24 -0800
* Subject: [PATCH net-next] hyperv: Implement netvsc_get_channels() ethool op
This adds support for reporting the actual and maximum combined channels count of the hv_netvsc driver via 'ethtool --show-channels'.
This required adding 'max_chn' to 'struct netvsc_device', and assigning it 'rsscap.num_recv_que' in 'rndis_filter_device_add'. Now we can access the combined maximum channel count via 'struct netvsc_device' in the ethtool callback.
Signed-off-by: Andrew Schwartzmeyer <andrew@schwartzmeyer.com>
---
 drivers/net/hyperv/hyperv_net.h | 1 +
 drivers/net/hyperv/netvsc_drv.c | 14 ++++++++++++++
 drivers/net/hyperv/rndis_filter.c | 6 +++++-
 3 files changed, 20 insertions(+), 1 deletion(-)
diff --git a/drivers/net/hyperv/hyperv_net.h b/drivers/net/hyperv/hyperv_net.h
index 384ca4f4de4a..4815843a6019 100644
--- a/drivers/net/hyperv/hyperv_net.h
+++ b/drivers/net/hyperv/hyperv_net.h
@@ -634,6 +634,7 @@ struct netvsc_device {
 
 	struct vmbus_channel *chn_table[NR_CPUS];
 	u32 send_table[VRSS_SEND_TAB_SIZE];
+	u32 max_chn;
 	u32 num_chn;
 	atomic_t queue_sends[NR_CPUS];
 
diff --git a/drivers/net/hyperv/netvsc_drv.c b/drivers/net/hyperv/netvsc_drv.c
index 15d82eda0baf..a06bd6614007 100644
--- a/drivers/net/hyperv/netvsc_drv.c
+++ b/drivers/net/hyperv/netvsc_drv.c
@@ -687,6 +687,19 @@ static void netvsc_get_drvinfo(struct net_device *net,
 	strlcpy(info->fw_version, "N/A", sizeof(info->fw_version));
 }
 
+static void netvsc_get_channels(struct net_device *net,
+				struct ethtool_channels *channel)
+{
+	struct net_device_context *net_device_ctx = netdev_priv(net);
+	struct hv_device *dev = net_device_ctx->device_ctx;
+	struct netvsc_device *nvdev = hv_get_drvdata(dev);
+
+	if (nvdev) {
+		channel->max_combined	= nvdev->max_chn;
+		channel->combined_count = nvdev->num_chn;
+	}
+}
+
 static int netvsc_change_mtu(struct net_device *ndev, int mtu)
 {
 	struct net_device_context *ndevctx = netdev_priv(ndev);
@@ -760,6 +773,7 @@ static void netvsc_poll_controller(struct net_device *net)
 static const struct ethtool_ops ethtool_ops = {
 	.get_drvinfo	= netvsc_get_drvinfo,
 	.get_link	= ethtool_op_get_link,
+	.get_channels = netvsc_get_channels,
 };
 
 static const struct net_device_ops device_ops = {
diff --git a/drivers/net/hyperv/rndis_filter.c b/drivers/net/hyperv/rndis_filter.c
index 7816d98bdddc..ca81de04bc76 100644
--- a/drivers/net/hyperv/rndis_filter.c
+++ b/drivers/net/hyperv/rndis_filter.c
@@ -1027,6 +1027,7 @@ int rndis_filter_device_add(struct hv_device *dev,
 
 	/* Initialize the rndis device */
 	net_device = hv_get_drvdata(dev);
+	net_device->max_chn = 1;
 	net_device->num_chn = 1;
 
 	net_device->extension = rndis_device;
@@ -1094,6 +1095,7 @@ int rndis_filter_device_add(struct hv_device *dev,
 	if (ret || rsscap.num_recv_que < 2)
 		goto out;
 
+	net_device->max_chn = rsscap.num_recv_que;
 	net_device->num_chn = (num_online_cpus() < rsscap.num_recv_que) ?
 			 num_online_cpus() : rsscap.num_recv_que;
 	if (net_device->num_chn == 1)
@@ -1140,8 +1142,10 @@ int rndis_filter_device_add(struct hv_device *dev,
 	ret = rndis_filter_set_rss_param(rndis_device, net_device->num_chn);
 
 out:
-	if (ret)
+	if (ret) {
+		net_device->max_chn = 1;
 		net_device->num_chn = 1;
+	}
 	return 0; /* return 0 because primary channel can be used alone */
 
 err_dev_remv:
-- 
2.3.0
===== set-channels =====
From aecd06e665323c07aae18ca9f394a4de282a12c3 Mon Sep 17 00:00:00 2001
From: Andrew Schwartzmeyer <andrew@schwartzmeyer.com>
Date: Mon, 27 Apr 2015 22:45:18 -0700
Subject: [PATCH net-next] hyperv: Implement netvsc_set_channels() ethtool op
* Remove and add RNDIS device with new channel count
* Added num_chn to struct device_info so it can be passed to
 rndis_filter_device_add()
* Initialize device_info to 0 always
* Set num_chan in rndis_filter_device_add() when passed via
 struct device_info, else use num_cpus()
* Add local u32 num_chan to rndis_filter_device_add() for shorter name
* Declare netvsc_link_change
* Check that NVSP version is at least 5
* Check given combined count against bounds [1, num_online_cpus()]
* Use num_online_cpus() if given count is too high (and log a notice)
* Check return values of netif set tx/rx queues
* Attempt to recover RNDIS if it fails with new setting
Signed-off-by: Andrew Schwartzmeyer <andrew@schwartzmeyer.com>
---
 drivers/net/hyperv/hyperv_net.h | 1 +
 drivers/net/hyperv/netvsc_drv.c | 126 ++++++++++++++++++++++++++++++++++++++
 drivers/net/hyperv/rndis_filter.c | 18 ++++--
 3 files changed, 140 insertions(+), 5 deletions(-)
diff --git a/drivers/net/hyperv/hyperv_net.h b/drivers/net/hyperv/hyperv_net.h
index a10b31664709..318f1f0fa1e6 100644
--- a/drivers/net/hyperv/hyperv_net.h
+++ b/drivers/net/hyperv/hyperv_net.h
@@ -162,6 +162,7 @@ struct netvsc_device_info {
 	unsigned char mac_adr[ETH_ALEN];
 	bool link_state;	/* 0 - link up, 1 - link down */
 	int ring_size;
+	u32 num_chn;
 };
 
 enum rndis_device_state {
diff --git a/drivers/net/hyperv/netvsc_drv.c b/drivers/net/hyperv/netvsc_drv.c
index a3a9d3898a6e..cfe6e8d59c3c 100644
--- a/drivers/net/hyperv/netvsc_drv.c
+++ b/drivers/net/hyperv/netvsc_drv.c
@@ -52,6 +52,8 @@ static int ring_size = 128;
 module_param(ring_size, int, S_IRUGO);
 MODULE_PARM_DESC(ring_size, "Ring buffer size (# of pages)");
 
+static void netvsc_link_change(struct work_struct *);
+
 static void do_set_multicast(struct work_struct *w)
 {
 	struct net_device_context *ndevctx =
@@ -729,12 +731,130 @@ static void netvsc_get_channels(struct net_device *net,
 	}
 }
 
+static int netvsc_set_channels(struct net_device *net,
+			 struct ethtool_channels *channel)
+{
+	struct net_device_context *net_device_ctx = netdev_priv(net);
+	struct hv_device *dev = net_device_ctx->device_ctx;
+	struct netvsc_device *nvdev = hv_get_drvdata(dev);
+	struct netvsc_device_info device_info;
+	const int num_cpus = num_online_cpus();
+	const int num_chn = nvdev->num_chn;
+	int ret;
+
+	if (!nvdev || nvdev->destroy)
+		return -ENODEV;
+
+	if (!channel) {
+		pr_warn("struct ethtool_channels was null\n");
+		return -EINVAL;
+	}
+
+	if (nvdev->nvsp_version < NVSP_PROTOCOL_VERSION_5) {
+		pr_info("vRSS unsupported before NVSP Version 5\n");
+		return -EINVAL;
+	}
+
+	if (channel->combined_count < 1)
+		return -EINVAL;
+
+	/* use max if combined count exceeds it */
+	if (channel->combined_count > num_cpus) {
+		pr_info("set channels too high, using %d\n", num_cpus);
+		channel->combined_count = num_cpus;
+	}
+
+	/* remove RNDIS */
+	nvdev->start_remove = true;
+	cancel_work_sync(&net_device_ctx->work);
+
+	netif_tx_disable(net);
+	rndis_filter_device_remove(dev);
+
+	/* set new num_chn */
+	nvdev->num_chn = channel->combined_count;
+
+	/* add RNDIS */
+	net_device_ctx->device_ctx = dev;
+	hv_set_drvdata(dev, net);
+
+	memset(&device_info, 0, sizeof(device_info));
+	device_info.num_chn = nvdev->num_chn;
+	device_info.ring_size = ring_size;
+
+	ret = rndis_filter_device_add(dev, &device_info);
+	if (ret != 0)
+		goto recover;
+
+	ret = netif_set_real_num_tx_queues(net, nvdev->num_chn);
+	if (ret != 0)
+		goto recover;
+
+	ret = netif_set_real_num_rx_queues(net, nvdev->num_chn);
+	if (ret != 0)
+		goto recover;
+
+	netif_tx_wake_all_queues(net);
+
+	return 0;
+
+ recover:
+
+	/* remove RNDIS */
+	nvdev->start_remove = true;
+	cancel_work_sync(&net_device_ctx->work);
+
+	netif_tx_disable(net);
+	rndis_filter_device_remove(dev);
+
+	/* set previous num_chn */
+	nvdev->num_chn = num_chn;
+
+	/* add RNDIS */
+	net_device_ctx->device_ctx = dev;
+	hv_set_drvdata(dev, net);
+
+	memset(&device_info, 0, sizeof(device_info));
+	device_info.num_chn = nvdev->num_chn;
+	device_info.ring_size = ring_size;
+
+	ret = rndis_filter_device_add(dev, &device_info);
+	if (ret != 0) {
+		netdev_err(net, "unable to add netvsc device (ret %d)\n", ret);
+		goto error;
+	}
+
+	ret = netif_set_real_num_tx_queues(net, nvdev->num_chn);
+	if (ret != 0) {
+		netdev_err(net, "could not set tx queue count (ret %d)\n", ret);
+		goto error;
+	}
+
+	ret = netif_set_real_num_rx_queues(net, nvdev->num_chn);
+	if (ret != 0) {
+		netdev_err(net, "could not set rx queue count (ret %d)\n", ret);
+		goto error;
+	}
+
+	netif_tx_wake_all_queues(net);
+
+	return 0;
+
+ error:
+	free_netdev(net);
+	hv_set_drvdata(dev, NULL);
+	netdev_err(net, "net device not enabled (ret %d)\n", ret);
+
+	return ret;
+}
+
 static int netvsc_change_mtu(struct net_device *ndev, int mtu)
 {
 	struct net_device_context *ndevctx = netdev_priv(ndev);
 	struct hv_device *hdev = ndevctx->device_ctx;
 	struct netvsc_device *nvdev = hv_get_drvdata(hdev);
 	struct netvsc_device_info device_info;
+
 	int limit = ETH_DATA_LEN;
 
 	if (nvdev == NULL || nvdev->destroy)
@@ -756,7 +876,10 @@ static int netvsc_change_mtu(struct net_device *ndev, int mtu)
 
 	ndevctx->device_ctx = hdev;
 	hv_set_drvdata(hdev, ndev);
+
+	memset(&device_info, 0, sizeof(device_info));
 	device_info.ring_size = ring_size;
+
 	rndis_filter_device_add(hdev, &device_info);
 	netif_tx_wake_all_queues(ndev);
 
@@ -803,6 +926,7 @@ static const struct ethtool_ops ethtool_ops = {
 	.get_drvinfo	= netvsc_get_drvinfo,
 	.get_link	= ethtool_op_get_link,
 	.get_channels = netvsc_get_channels,
+	.set_channels = netvsc_set_channels,
 };
 
 static const struct net_device_ops device_ops = {
@@ -876,6 +1000,8 @@ static int netvsc_probe(struct hv_device *dev,
 	int ret;
 	u32 max_needed_headroom;
 
+	memset(&device_info, 0, sizeof(device_info));
+
 	net = alloc_etherdev_mq(sizeof(struct net_device_context),
 				num_online_cpus());
 	if (!net)
diff --git a/drivers/net/hyperv/rndis_filter.c b/drivers/net/hyperv/rndis_filter.c
index 0d92efefd796..85091333c4c2 100644
--- a/drivers/net/hyperv/rndis_filter.c
+++ b/drivers/net/hyperv/rndis_filter.c
@@ -1011,7 +1011,7 @@ int rndis_filter_device_add(struct hv_device *dev,
 	unsigned long t;
 	struct ndis_recv_scale_cap rsscap;
 	u32 rsscap_size = sizeof(struct ndis_recv_scale_cap);
-	u32 mtu, size;
+	u32 mtu, size, max_chn;
 
 	rndis_device = get_rndis_device();
 	if (!rndis_device)
@@ -1096,12 +1096,19 @@ int rndis_filter_device_add(struct hv_device *dev,
 	ret = rndis_filter_query_device(rndis_device,
 					OID_GEN_RECEIVE_SCALE_CAPABILITIES,
 					&rsscap, &rsscap_size);
-	if (ret || rsscap.num_recv_que < 2)
+	max_chn = rsscap.num_recv_que;
+	if (ret || max_chn < 2)
 		goto out;
 
-	net_device->max_chn = rsscap.num_recv_que;
-	net_device->num_chn = (num_online_cpus() < rsscap.num_recv_que) ?
-			 num_online_cpus() : rsscap.num_recv_que;
+	net_device->max_chn = max_chn;
+
+	if (device_info->num_chn && device_info->num_chn < max_chn) {
+		net_device->num_chn = device_info->num_chn;
+	} else {
+		net_device->num_chn = (num_online_cpus() < max_chn) ?
+				 num_online_cpus() : max_chn;
+	}
+
 	if (net_device->num_chn == 1)
 		goto out;
 
@@ -1109,6 +1116,7 @@ int rndis_filter_device_add(struct hv_device *dev,
 					 NETVSC_PACKET_SIZE);
 	if (!net_device->sub_cb_buf) {
 		net_device->num_chn = 1;
+		/* should we set max_chn here too? */
 		dev_info(&dev->device, "No memory for subchannels.\n");
 		goto out;
 	}
-- 
2.4.0
====Expo Deliverables====
Here are the following deliverables for the 22nd Annual Senior Design Expo:
Senior Design Expo Deliverables	Senior Design Expo Deliverables 	Senior Design Expo Deliverables
Document 	Description 	Link
Poster 	Display booth project design poster. 	
Technical Presentation Slides 	Powerpoint slides used for the expo technical presentation.	
==Team Biographies==
===Andrew Schwartzmeyer===
Computer Science and Mathematics Student
Hometown: Seattle, WA
Bio: Andrew is a senior undergraduate student in Computer Science and Mathematics at the University of Idaho. He is the current chair of the University of Idaho Association of Computing Machinery, and an avid open source developer. He blogs at schwartzmeyer.com, and is interested in Linux systems programming, evolutionary computation, security and cryptography, server administration automation (with Puppet), and functional programming.
Email: schw2620@vandals.uidaho.edu
===Justin Hall===
Computer Science Student
Hometown: Lewiston, ID
Bio: Justin is a senior in Computer Science at the University of Idaho, currently pursuing his Bachelors of Science. He spends summers as a freelance front-end Javascript developer, particularly the AngularJS framework. His interests focus on front-end and full-stack web development. 
Email: hall5714@vandals.uidaho.edu
===Keith Drew===
Computer Science Student
Hometown: Olympia, Washington
Bio: Keith is a senior undergraduate student in Computer Science at the University of Idaho. He is also a NSF Scholarship for Service (SFS) CyberCorps Student. He is interested cyber security, specifically secure programming. 
Email: keithd@vandals.uidaho.edu
===Eric Doman===
Computer Science Student
Hometown: St. Maries, Idaho
Bio: Eric is a senior undergraduate student in Computer Science at the University of Idaho. He is a member of the University of Idaho Association of Computing Machinery. He is interested in computer graphics and visualization. In his free time, Eric enjoys running and playing soccer. 
Email: doma8101@vandals.uidaho.edu
===Chris Waltrip (Fall 2014)===
Computer Science Student
Hometown: Coeur d'Alene, ID
Bio: Chris is a senior undergraduate in Computer Science at the University of Idaho. He is also a NSF Scholarship for Service (SFS) CyberCorps Student and system administrator for the Center for Secure and Dependable Systems (CSDS). His interests lie in information assurance and network security.
Email: walt2178@vandals.uidaho.edu
==Appendices==
===References===
===Other Links===
* Linux Kernel in a Nutshell
* Linux Kernel Coding Style
* netdev FAQ
* Linux Kernel Hacking
* Guide to building the Linux kernel
* Linux Kernel Debugging
* Linux Kernel Hacking Tools
* Build Your Own Kernel Modules