Windows VMs under Linux with Native Performance - Page 1
Forum section: electronics / general

author: gnif
author_rank: Unknown
timestamp: August 06, 2022, 09:10:30 pm
content: Hi All,This is a shameless plug for the free open source project I wrote which I use on a daily basis along with thousands of others that allows the use of a Windows VM in Linux that has a VFIO/SR-IOV or GPU pass-through setup.2 Minute Demo (aka, TL;DR; version)https://youtu.be/7XbQOjfnxbUBefore I begin I should explain what this is as it's not a commonly known technology and as such many people do not know that it's even possible. With the advent of cpu extensions like IOMMU and VFIO we are able to isolate PCIe devices and give them to a virtual machine for it's sole use. This technology has been used for years now in the hosting industry where network interface cards support advanced features like SR-IOV/VFIO, allowing them to be split into multiple "virtual devices", or more accurately known as "virtual functions". This is done to give the VM direct access to the hardware so that it can obtain bare metal performance on the hardware.For a while now there has been a growing community of users that are using this to instead pass a complete GPU to the Virtual Machine, allowing the VM to have bare metal 3D rendering performance. This does require you to have two GPUs in your system, either by installing another one, or if you are using a laptop you may already have two GPUs (iGPU + AMD/NVidia something). This is commonly known as a VFIO Pass-through configuration, or VGA Pass-through configuration. There is one shortcoming of this type of a setup, the GPU has no idea that it's inside a VM and as such still wishes to output to a physical monitor connected to it's output.This is where my software comes along (Looking Glass -https://looking-glass.io). This is a two part application that makes use of a special virtual device called IVSHMEM, or "Inter-Virtual Shared Memory" to map a block of shared RAM into the VM that can be used to move mass amounts of data with extremely low latency in/out of the VM. The "host" application (not the host system, yes the naming is confusing), runs inside the guest and captures the frame-buffer output of the GPU and feeds it into the shared memory segment.The client side application that either runs on the host, or even in another VM (yes, we can run VM to VM) then takes the feed in shared RAM and is essentially a high performance RDP client but instead of using a slow network protocol with compression, it's lossless and the latency is outstanding. In-fact in some cases we can get the frames to screen faster then the GPU will output them to the physical monitor as the Linux graphics pipeline is far shorter then Windows.So you might be now asking what kind of crazy exotic hardware you need for this kind of a setup? That's just it, none. You can do this on any CPU that has a decent core count (6+) and use any secondary GPU you have (provided it plays nice with VFIO, some AMD GPUs have hardware bugs that prevent their use). We have people running this on 5 year old laptops, in use in universities by students and professors for comp-sci projects, and artists that need access to windows only applications such as the Adobe suite, AutoCad, etc.Personally I have this working on my Intel laptop, a Ryzen 7 1700x desktop, and my current workhorse that was kindly donated for this work, an AMD EPYC Milan system. I have not needed or even wanted to dual boot my Linux system in 5 years now, either for gaming or productive usages.One additional feature I have not mentioned yet is for those that want to capture the VM for streaming/recording, etc. I have also written a native OBS plugin for Looking Glass that allows you to take the feed directly into OBS with no additional overheads. This then allows you to offload the video processing and encode to the host system (or another VM), even offloading to a GPU with hardware encode capability.

author: Halcyon
author_rank: Unknown
timestamp: August 07, 2022, 06:13:08 am
content: Brilliant work as always Geoffrey. I've plugged you on Linkedin.

author: PKTKS
author_rank: Super Contributor
timestamp: August 07, 2022, 07:07:06 pm
content: Passing here to drop the brilliant work mention as well...But also to remind that GPU pass and virtIO VMs require a proper kernel compilation.The options should allow the PCI and IOMMU specific options for that to happen.Indeed an awesome step aheadPaul

author: gnif
author_rank: Unknown
timestamp: August 07, 2022, 08:45:12 pm
content: Quote from: PKTKS on August 07, 2022, 07:07:06 pmWhat do you mean? every major distro's stock kernel supports this and has done so for the past 5 years now.

author: bd139
author_rank: Super Contributor
timestamp: August 07, 2022, 10:21:37 pm
content: PKTKS is about 20 years behind the curve based on his other threadsInteresting idea though. Will read into this tomorrow when I get some time. Either way more effort on this is appreciated so nice job

author: abquke
author_rank: Regular Contributor
timestamp: August 07, 2022, 10:29:55 pm
content: Can windows programs in VMs talk to hardware devices yet?Software guys just dropped on me that they couldn't get Wine to use vendor software to talk to their USB widget... but we can't use windows because they couldn't access an I2C bus to control GPIO in windows...

author: bd139
author_rank: Super Contributor
timestamp: August 07, 2022, 10:32:14 pm
content: You can with virtualbox and the non free extensions from oracle. I was using it to do STM32 flash ages ago.

author: gnif
author_rank: Unknown
timestamp: August 07, 2022, 10:36:29 pm
content: Quote from: abquke on August 07, 2022, 10:29:55 pmThat's exactly what VFIO is.Quote from: abquke on August 07, 2022, 10:29:55 pmWe commonly either pass-through a USB device to the VM (works mostly) or for a more reliable & complete solution you can pass through one of your USB controllers. If you don't have an extra controller you can pass through you can install a PCIe one and pass that into the VM. Controller pass-through gives 100% compatibility and native performance.Quote from: bd139 on August 07, 2022, 10:32:14 pmVirtualBox today is way behind the ball though, i'd suggest you use LibVirt which at it's core is QEMU/KVM and has far better supportAs for STM32 flashing, you can just use the other FOSS project I startedhttps://sourceforge.net/p/stm32flash/wiki/Home/It's packaged in most distros these days also.

author: Ed.Kloonk
author_rank: Super Contributor
timestamp: August 07, 2022, 10:50:25 pm
content: When I were a younger man, and virtualization was just a itch in Andy Groves' pants, I understood that virt would be good for sandboxing and security. I don't do gaming nor windows, but I don't understand how you could allow it access to the hardware and still trust it not betray your trust.

author: gnif
author_rank: Unknown
timestamp: August 08, 2022, 03:04:05 am
content: Quote from: Ed.Kloonk on August 07, 2022, 10:50:25 pmRead up on IOMMU, this isolates the hardware from your system preventing it from being able to compromise the system. Not only does it protect your system from malicious devices, it also can be used to protect your system from a malicious virtual machine that has access/use of the physical device. Under KVM it is a requirement to use IOMMU due to the obvious security concerns of running without it.

author: PKTKS
author_rank: Super Contributor
timestamp: August 08, 2022, 08:48:34 am
content: Quote from: gnif on August 07, 2022, 08:45:12 pmI have stopped using so called stock kernels way back on mid 00s..After a decade+ using these stock kernels and off the shelf distros.. it became an obvious problem that server setups have nothing in common with EDA and DAW workstations... and even less in common with gaming.. and absolute nothing with proxys..So ... i made forks from LFS with my required specs.. things are several orders of magnitude SAFER FAST and smaller in size..So. unless you plan to run a bloated messy crammed stuff.. you want a clean kernel..My nephew game box runs stock stuff..Not my servers or stations... decades for now.Can not stress how important it isYou can only realize that after having them on hand for own comparisonPaul

author: gnif
author_rank: Unknown
timestamp: August 08, 2022, 08:55:29 am
content: Quote from: PKTKS on August 08, 2022, 08:48:34 amSo you have a low latency audio requirement, which is a realtime task, which requires a low latency kernel which impacts power usage and overall system throughput for most other tasks... got it.Quote from: PKTKS on August 08, 2022, 08:48:34 amOk? so even if it's only one order of magnitude faster, that must mean a 3GHz system on your kernel would be equivalent to a 30GHz system? You can't make such broad claims... some aspect of your system for your use may be faster, but obviously not the entire system. As for size, so what? Size is only an issue if you're running on a MCU with limited memory.Quote from: PKTKS on August 08, 2022, 08:48:34 amDefine clean? The kernel modules are only loaded as needed to support your hardware.Quote from: PKTKS on August 08, 2022, 08:48:34 amNo idea what this has to do with any of this.Quote from: PKTKS on August 08, 2022, 08:48:34 amYou're preference, but not a requirement by any means. Again, also if you're running servers tuned for low latency/realtime type tasks, your servers overall throughput for all other tasks will be hampered.Quote from: PKTKS on August 08, 2022, 08:48:34 amSorry, but I disagree to a large extent, please see/use Arch if you want to control every aspect of your system.When it comes to running visualised environments for tasks as this thread is discussing your "faster", low latency kernels actually cause big issues due to the guest being starved of CPU time when it needs to service hardware interrupts.Please be aware that I am actually an active kernel developer working on KVM and QEMU, along with the audio subsystems for Linux. With a stock kernel we can achieve latency numbers lower then windows running native on bare metal. At the end of the day it comes down to your use case and demands of your workload.

author: PKTKS
author_rank: Super Contributor
timestamp: August 08, 2022, 09:00:05 am
content: Exactly.What the point of running qemu on a proxy?Or having these options, IOMMU virtfs, in kernel running a dedicated workstation without qemu?None.It is useless and security hasslePaul

author: gnif
author_rank: Unknown
timestamp: August 08, 2022, 09:06:35 am
content: Quote from: PKTKS on August 08, 2022, 09:00:05 amIOMMU protects your system from a faulty device that decides to use DMA to read/write RAM and crash your entire system. It applies to desktop and server environments and is not limited to virtualisation usage. Also it's a hardware feature of your CPU, not a kernel feature, you always have it, the kernel just needs to program the hardware so you get the free (no overheads, no latency increases) protection it offers.Quote from: PKTKS on August 08, 2022, 09:00:05 amHuh? simple, don't install qemu

author: PKTKS
author_rank: Super Contributor
timestamp: August 08, 2022, 09:11:37 am
content: Actually it forces you to enable and compile a whole subset of options. .Only usable while running a VMAll other setups eg not VM, do not require this.. and require other fine tunes..Probably one will only realize that by the moment you need one feature and not the others..Paul

author: gnif
author_rank: Unknown
timestamp: August 08, 2022, 09:16:33 am
content: Quote from: PKTKS on August 08, 2022, 09:11:37 amIncorrect, the IOMMU is configured very early in the system boot by the kernel to provide device isolation and system protection, and it does not force you to use any VM options at all:Code:CONFIG_IOMMU_SUPPORT: IOMMU Hardware SupportGeneral informationsThe Linux kernel configuration item CONFIG_IOMMU_SUPPORT:prompt: IOMMU Hardware Supporttype: booldepends on: CONFIG_MMUdefined in drivers/iommu/Kconfigfound in Linux kernels: 3.1–3.19, 4.0–4.20, 5.0–5.19, 5.19+HEADThe only dependency is to have a MMU, which is a Linux kernel requirement.IOMMU is not strictly a VM feature, but it can be leveraged in such configurations for added security.QuoteSource:https://en.wikipedia.org/wiki/Input%E2%80%93output_memory_management_unitDisabling IOMMU on modern systems hurts performance AND system stabillity/security.Edit: Don't confuse IO virtualization with Virtual Machines, they are very different things.Edit2: Consider a device with a binary blob that needs loading to run it, lets say a Intel NIC, or a NVidia/AMD GPU which have closed firmware/drivers. IOMMU stops someone using the device as a mechanism to bypass system security either through intentional backdoors in the firmware/hardware, or bugs. There is no way in hell i'd ever consider putting a server online without IOMMU support enabled.

author: rob77
author_rank: Super Contributor
timestamp: August 08, 2022, 10:23:53 am
content: Quote from: PKTKS on August 08, 2022, 08:48:34 amnot using stock kernels coming with the distro made sense in the 90's but not in mid 00's. how do you handle security updates ?i remember kernels with major number 0 , recompiling kernels was a pretty common routine back in the days but it simply makes no sense nowadays.unless you develop an embedded system there is no reason to compile your kernel.. speed gain is next to none and security wise you put yourself into worse situation than using the stock kernel coming with the distro.

author: bd139
author_rank: Super Contributor
timestamp: August 08, 2022, 10:28:41 am
content: PKTKS doesn't exist on the same planet as the rest of us.

author: gnif
author_rank: Unknown
timestamp: August 08, 2022, 11:04:47 am
content: Yeah, I had this also when I had an Athlon x64 and I wanted to experiment with the x86_64 support. I also had an EPOCH motherboard with the nForce2 chipset, which the kernel had no support for, but there were patches.These days I only compile the kernel if I am doing development work on it, most of which is latency related testing or KVM debugging. Fixing AMDs bugs for them has also been a major cause of kernel re-compiles too, but to save time/effort I often resort to things like ftrace/kprobes for poking at PCI devices that are misbehaving.

author: PKTKS
author_rank: Super Contributor
timestamp: August 08, 2022, 01:07:35 pm
content: There seems to be a misunderstanding ...Combined with a necessary rant to insult and snob others (me included..)I *ALWAYS* configure theKERNELoption IOMMU set as stated from above.IT IS A NECESSARY COMPILE TIME OPTION SINCE version 2.x something.However..Please give me a break and just realize thatGPU passthrough technology (presenting PCI GPU to a virtual machine) is also often known as IOMMU, although this is a bit of a misnomer, since the IOMMU is the hardware technology that provides this feature but also provides other features such as some DMA attacks and etc_etall..Sooooo GPU passthrough as intended (IOMMU) and libvirt and friends are related to a QEMU CLIENT machine and will require a proper HOST KERNEL .. to enable PCI passthrough IOMMU..All options related to properly set the PCI passthrough IOMMUDON NOT MAKE MISTAKES ABOUT these 2 things... and start this sick insulting show...Hope to shutup this nasty sick insults about things not related and degenerated...I have reached such that age in life where insulting ... or having such freaking time with others just do not add anything to me... I am a serious and very focused in intentions...So spare me this nasty things... please.Paul

author: gnif
author_rank: Unknown
timestamp: August 08, 2022, 01:19:28 pm
content: Quote from: PKTKS on August 08, 2022, 01:07:35 pmActually IOMMU is never called or confused with Passthrough, the term you're looking for is VFIO (Virtual Function IO) which makes use of IOMMU to map the device into a protective region to prevent a rogue VM from breaking/crashing or exploiting the host (VM escape). I mean, the kernel module use for this is literally called `vfio_pci`. There is a reddit community for VFIO of over 33k users (https://www.reddit.com/r/VFIO/), and a Discord community called VFIO of over 4.6k users. Futher to this, VFIO is not limited to use in Virtual Machines, it has user-space access applications bypassing the kernel in special circumstances with custom devices.You can actually perform VFIO pass-through without IOMMU support at all, but it is dangerous to do so and as such by default the KVM kernel module will refuse to do so unless patched. For example, the ACS override patch fools the kernel into believing that the PCIe devices are all in isolated IOMMU groups when they may not be.As for your argument of disabling these features to make a kernel faster or more secure, this is just simply not true. Kernel modules are only loaded as they are needed either by some feature you want (like KVM) or a device that requires the module to operate, otherwise they sit idle on disk. The VM features exist in the `kvm` kernel module and all major distros compile these as modules, they are not inlined into the kernel. Unless you have actually modprobed these kernel modules then you have a clean kernel that is not full of bloat you don't need.From a security point of view, If an attacker on your system has the permissions needed to insmod/modprobe a kernel module, this means they have `root` access and it doesn't matter what kernel you run, or how you built it, your security has already been defeated.

author: PKTKS
author_rank: Super Contributor
timestamp: August 08, 2022, 01:22:46 pm
content: Quote from: gnif on August 08, 2022, 01:19:28 pmAs you wish my dear... by now you just got the misunderstanding ..I have no intention to lecture insult or or go further..Pretty sure you got the point between the 2 uses of IOMMU...Paul

author: krokodyl1220
author_rank: Contributor
timestamp: August 10, 2022, 03:23:56 pm
content: Ranting and discussion aside, I am another one of happy people using LookingGlass. In my case it is to run ECAD and MCAD and it works flawlessly. Thank you!Recently I have created a simple script to help when using VMs in user session mode (I dislike running VMs as root on my workstation) to isolate the CPUs with cgroups. If you want to check it out see:https://gitlab.com/krokodyl1220/vfio-vm-toolsUnfortunately for PKTKS it uses systemd

author: Marco
author_rank: Super Contributor
timestamp: August 10, 2022, 05:11:11 pm
content: How much memory do you use in the laptop to get decent performance?

author: gnif
author_rank: Unknown
timestamp: August 10, 2022, 05:24:12 pm
content: Quote from: Marco on August 10, 2022, 05:11:11 pmThe laptop has 16GB, which I split in two and give 8GB to the VM.

author: RoGeorge
author_rank: Super Contributor
timestamp: August 10, 2022, 06:56:21 pm
content: First of all wow, great project!Quote from: gnif on August 06, 2022, 09:10:30 pmSecond, it just happens that today I'm reinstalling all from scratch (posting from a USB live Devuan now), and would be very nice to add looking glass.I've briefly looked at thehttps://looking-glass.io/docs/B5.0.1/faq/and still have questions.Last time I've tried virtualization, couldn't give the video card to a guest machine no matter what. The closest I could come was with proxmox (a bare-metal type hypervisor), could pass the GPU, but couldn't install the video drivers in the Windows guest machine.IIRC it was Error 43 from nVidia driver installer, and after searching about that, it turned out that's an intentional software limitation included by nVidia in their Windows drivers and in the firmware of the videocard. The only card allowed to passthrough were the very expensive ones, like the Quadro line. Other cards would have been able to work, unless they were limited by nVidia (for some cards there are even hardware hacks to make the chip present itself as a Quadro).Would Looking Glass work for a consumer grade nVidia (GeForce GTX760 2GB video RAM + i7 with a second Intel GPU)?

author: krokodyl1220
author_rank: Contributor
timestamp: August 10, 2022, 07:23:01 pm
content: If you have problems with code 43 there are workarounds, but some months ago Nvidia started supporting vfio out of the box without code 43 on all cards, also consumer. I use GTX 3050 but also successfully used GTX 1050 and Intel integrated GPU with gvt-g.As far as I can see the GTX 760 is on driver 473 and Nvidia allowed VMs from driver 465, so it should work out of the box

author: gnif
author_rank: Unknown
timestamp: August 10, 2022, 09:27:20 pm
content: Quote from: krokodyl1220 on August 10, 2022, 07:23:01 pmThere are exceptions to this.#1. Laptop NVidia GPUs in some instances require the commonly used and known workaround.#2. If you want to use custom resolutions the native nvidia "support" locks you out of this feature unless you also use the workaround.#3. AMD drivers also require this for Vega or later generation GPUs due to a bug which also causes the Code 43 error.The workaround is to hide the KVM CPUID leaf (not disable these features), and change the KVM vendor_id to something random.https://passthroughpo.st/apply-error-43-workaround/

author: krokodyl1220
author_rank: Contributor
timestamp: August 11, 2022, 06:56:18 am
content: Yes, you are right. I was just replying to RoGeorge.

author: gnif
author_rank: Unknown
timestamp: August 11, 2022, 07:04:27 am
content: If you have issues again@RoGeorgejump into the VFIO discord, there are tons of very smart people there that are very willing to help you (myself included)https://discord.gg/f63cXwH

author: RoGeorge
author_rank: Super Contributor
timestamp: August 11, 2022, 10:05:28 am
content: Thank you for the invitation, you are too kind. Sorry for not taking advantage of it.In general I won't go on a platform that requires registration to read. Not a fan of any chat oriented platforms, either.So far didn't try installing Looking Glass, still working at a clean install with root ZFS across all OSs. That would probably take many days, since I want to have at least a FreeBSD as a fallback, a Gentoo or Funtoo (something built from sources but not LFS, that would be too demanding to maintain alone as a desktop), maybe a Devuan and an Ubuntu, too, not sure yet.Meanwhile searched again about virtualization in general, and it seems nVidia got more relaxed about that starting last year:https://nvidia.custhelp.com/app/answers/detail/a_id/5173That makes me hope Looking Glass will work with my hardware, too.

author: gnif
author_rank: Unknown
timestamp: August 11, 2022, 10:08:56 am
content: Quote from: RoGeorge on August 11, 2022, 10:05:28 amNo worries mate, but to clarify, your hardware always would have worked, it was a simple matter of VM configuration. You do not need, nor have you ever needed a Quadro/Tesla, etc for this.As for being a chat platform, we use it because we render real time live support, there is no other real way to do this.

author: RoGeorge
author_rank: Super Contributor
timestamp: August 11, 2022, 11:40:05 am
content: Found this video of yours for how the Looking Glass works:https://youtu.be/U44lihtNVVMIf I got it correctly, the main idea is to use what in an FPGA would have been called a dual-port RAM, here a physical RAM region, addressed once from the guest machine and once from the host machine, so to transfer the video buffer from one OS to the other, except here the dual addressability is implemented in software (while in an FPGA interlocking would have been handled in hardware), yet the RAM buffer is still using DMA transfers to stay fast.Very clever!

author: gnif
author_rank: Unknown
timestamp: August 11, 2022, 12:33:29 pm
content: Quote from: RoGeorge on August 11, 2022, 11:40:05 amPretty much, however if your host GPU is Intel or AMD (or NVidia once the next FOSS driver drops) you can also use GPU DMA xfers from system ram to GPU ram offloading some of the workload from the CPU

author: RoGeorge
author_rank: Super Contributor
timestamp: August 12, 2022, 05:56:16 pm
content: Just FIY, was curious to try LookingGlass and followed the install page:https://looking-glass.io/docs/B5.0.1/build/#buildingFor my desktop, which is a former Kubuntu 20.04 LTS migrated to 22.04 LTS (Jammy Jellyfish), some additional packages were needed to compile Looking Glass, apart from the ones listed in the instructions:Code:# additional packages needed to compile, aside from the ones listed in instructionssudo apt install libpipewire-0.3-dev libpulse-dev libsamplerate0-devThe client built just fine (with the default options).Will try again this weekend to passthrough the nVidia GTX760, to test Looking Glass.

author: gnif
author_rank: Unknown
timestamp: August 12, 2022, 06:02:41 pm
content: Quote from: RoGeorge on August 12, 2022, 05:56:16 pmThis is because you didn't build the current "stable" release of B5.0.1, but rather bleeding edge.Documentation for Bleeding Edge is at:https://looking-glass.io/docs/bleeding

author: gnif
author_rank: Unknown
timestamp: August 13, 2022, 09:58:28 am
content: Quote from: evb149 on August 13, 2022, 09:54:10 amFirstly wow, talk about a wall of text... most of your questions can be answered by just some simple googling.Secondly open a new thread and don't hijack this one please.

author: gnif
author_rank: Unknown
timestamp: August 13, 2022, 12:06:22 pm
content: Quote from: evb149 on August 13, 2022, 11:28:47 amYou have asked a ton of questions all at once that do not relate directly to "Windows VMs under Linux with Native Performance". They are generic questions with regards to the state of virtualisation under Linux and the various technologies/hardware options.Quote from: evb149 on August 13, 2022, 11:28:47 amI did not mean to imply you have not done your research nor do I think your questions are stupid at all, they are just way left field of what this thread is about. You said it yourself, "very niche" questions, and they deserve their own thread where I would be happy to assist/help, etc.Quote from: evb149 on August 13, 2022, 11:28:47 amWindows VMsunder Linux with Native PerformanceThis is under the general computers segment of this forum. Please review the kinds of topics that go here before you question the validity of this thread existing here.Quote from: evb149 on August 13, 2022, 11:28:47 amThere are solutions without adding an extra GPU, you do not need to use Looking Glass, and I would be happy to discuss all of this in a thread where it is the topic of discussion.Quote from: evb149 on August 13, 2022, 11:28:47 amI have no reason to advertise my project, the goal was to share with people what is possible and help people to find Linux more useful/comfortable.

author: krokodyl1220
author_rank: Contributor
timestamp: September 11, 2022, 08:25:27 am
content: @gnif - somewhere in the project docs there was info that the virtual / emulated display to get rid of the need for dummy plugs was abandoned. Do you have any plans regarding this dummy driver?

author: gnif
author_rank: Unknown
timestamp: September 11, 2022, 08:27:56 am
content: Quote from: krokodyl1220 on September 11, 2022, 08:25:27 amIt was never even started, all that was done was some initial proof of concept work. It's still planned if enough funding can be raised to cover my development time & costs.See:https://github.com/sponsors/gnif

author: krokodyl1220
author_rank: Contributor
timestamp: September 11, 2022, 08:38:48 am
content: Oh ok, that wasn't clear, thanks for reply. Sometime ago I have stumbled upon this dummy project, perhaps it can be of some help with starting it at all?https://github.com/roshkins/IddSampleDriver

author: gnif
author_rank: Unknown
timestamp: September 11, 2022, 08:45:41 am
content: Quote from: krokodyl1220 on September 11, 2022, 08:38:48 amIt's actually what was used as the proof of concept

author: krokodyl1220
author_rank: Contributor
timestamp: September 11, 2022, 08:53:05 am
content: Hah, no further comments from me then