Portable Hotplugging
A Peek into NetBSD’s uvm_hotplug(9) API Development

Santhosh N. Raju
santhosh.raju@gmail.com

Cherry G. Mathew
cherry@NetBSD.org

September 23, 20171

Setting Expectations

What “Will” and “Will not” Be Covered?

2What “Will” and “Will not” Be Covered?
What will NOT be covered...
• Usage of uvm_hotplug(9)• Application of uvm_hotplug(9)• Refer man page of uvm_hotplug(9) for that2

What “Will” and “Will not” Be Covered?
What will NOT be covered...
• Usage of uvm_hotplug(9)• Application of uvm_hotplug(9)• Refer man page of uvm_hotplug(9) for that
So what I am going to talk about...
• Using TDD and how it was applied to uvm_hotplug(9)API• Design changes in uvm_hotplug(9) and how they were
implemented• Some interesting edge cases in uvm_hotplug(9)development• How we used atf(7) to do performance testing2

BackgroundThe Old Implementation• Uses a static array (vm_physmem[]) to hold segments• Maximum size of this array is defined in the macro
VM_PHYSSEG_MAX

3

The Old Implementation• Uses a static array (vm_physmem[]) to hold segments• Maximum size of this array is defined in the macro
VM_PHYSSEG_MAX• Implementation can be seen in uvm_page.cs t r u c t vm_physseg vm_physmem [VM_PHYSSEG_MAX ] ;i n t vm_nphysseg = 0 ;# d e f i n e vm_nphysmem
vm_nphysseg

We trace our steps into showing you how we converted thisarray implementation to a rbtree(3) based implementation.

3

Sanitising for uvm_hotplug(9)...Without loosing sanity

It took more than one step...
• Creating a reference API

4

...Without loosing sanity

It took more than one step...
• Creating a reference API• Separating out the existing API

4

...Without loosing sanity

It took more than one step...
• Creating a reference API• Separating out the existing API• Exposing the now separated API

4

...Without loosing sanity

It took more than one step...
• Creating a reference API• Separating out the existing API• Exposing the now separated API• Testing the API in userspace4Creating the Reference API• There were no Tests to use as a reference• We created an Idealised API to represent how the hotplug
API should look.
• Idealised API now acted as the baseline for the ATF tests
that should have been present in uvm(9)• Chuck Silvers gave valuable feedback when we weremaking this Idealised API

5Creating the Reference API• There were no Tests to use as a reference• We created an Idealised API to represent how the hotplug
API should look.
• Idealised API now acted as the baseline for the ATF tests
that should have been present in uvm(9)• Chuck Silvers gave valuable feedback when we weremaking this Idealised API• NOTE: The “Idealised” API was not a part of the NetBSD
build system. However the tests were buildable with
atf(7)

5

Separating the Existing API• Going through code mostly in uvm_page.c and some MD
parts.
• Separated stuff into uvm_physseg.c anduvm_physseg.h• Retrofitted relevant parts into various sections of IdealisedAPI

6

Exposing the new API• Kept structures that need not be exposed globally to theusers in a uvm_physseg.c file
• The uvm_physseg.h file nicely exposes all the “valid”
operations that can be done on the various opaquestructures that is used in this API• Exposed these utility functions via header file7

Exposing the new API• Kept structures that need not be exposed globally to theusers in a uvm_physseg.c file
• The uvm_physseg.h file nicely exposes all the “valid”
operations that can be done on the various opaquestructures that is used in this API• Exposed these utility functions via header file• This refactoring effort resulted in actual buildable andbootable code7

Testing in Userspace
Getting the kernel code to work in userspace8

Testing in Userspace
Getting the kernel code to work in userspace• Included the uvm_physseg.c file as part of the ATF test

8

Testing in Userspace
Getting the kernel code to work in userspace• Included the uvm_physseg.c file as part of the ATF test
• Stubbed / Re-implemented kernel API calls• Stubbed / Re-implemented dependent API calls

8Testing in Userspace
Getting the kernel code to work in userspace• Included the uvm_physseg.c file as part of the ATF test
• Stubbed / Re-implemented kernel API calls• Stubbed / Re-implemented dependent API calls• This is similar to Mocking APIs

8

Testing in Userspace
Getting the kernel code to work in userspace• Included the uvm_physseg.c file as part of the ATF test
• Stubbed / Re-implemented kernel API calls• Stubbed / Re-implemented dependent API calls• This is similar to Mocking APIs
An example of kmem_alloc() being stubbed
void ∗
kmem_alloc ( s i z e _ t s i z e , k m _ f l a g _ t f l a g s ){return malloc ( size ) ;}8Design and Implementation

From Static to Dynamic

We went for R-B Tree as the data structure for dynamic
opetations of insertion and deletion of memory segments.

9

From Static to Dynamic

We went for R-B Tree as the data structure for dynamic
opetations of insertion and deletion of memory segments.
• Implemented using the rbtree(3) part of NetBSD C
Library.

9

From Static to Dynamic

We went for R-B Tree as the data structure for dynamic
opetations of insertion and deletion of memory segments.
• Implemented using the rbtree(3) part of NetBSD C
Library.
• No worries about maintaining a sorted order. Made easierby RB_TREE_FOREACH()

9

From Static to Dynamic

We went for R-B Tree as the data structure for dynamic
opetations of insertion and deletion of memory segments.
• Implemented using the rbtree(3) part of NetBSD C
Library.
• No worries about maintaining a sorted order. Made easierby RB_TREE_FOREACH()• No more multiple strategies for maintaining the segments

9

From Static to Dynamic

We went for R-B Tree as the data structure for dynamic
opetations of insertion and deletion of memory segments.
• Implemented using the rbtree(3) part of NetBSD C
Library.
• No worries about maintaining a sorted order. Made easierby RB_TREE_FOREACH()• No more multiple strategies for maintaining the segments• Less code clutter

9

From Static to Dynamic

We went for R-B Tree as the data structure for dynamic
opetations of insertion and deletion of memory segments.
• Implemented using the rbtree(3) part of NetBSD C
Library.
• No worries about maintaining a sorted order. Made easierby RB_TREE_FOREACH()• No more multiple strategies for maintaining the segments• Less code clutter• Neater and cleaner API, compared to queue(3) andtree(3)9

Design Challenges• Handle for accessing segment changed between static
array and R-B Tree.

10

Design Challenges• Handle for accessing segment changed between static
array and R-B Tree.
• Index of array vm_physmem[] vs Pointer to structvm_physseg

10Design Challenges• Handle for accessing segment changed between static
array and R-B Tree.
• Index of array vm_physmem[] vs Pointer to structvm_physseg• Modifying a fundamental part of the operating system
implies every single architecture port of NetBSD isaffected. (78 at the time of writing this)10Design Challenges• Handle for accessing segment changed between static
array and R-B Tree.
• Index of array vm_physmem[] vs Pointer to structvm_physseg• Modifying a fundamental part of the operating system
implies every single architecture port of NetBSD isaffected. (78 at the time of writing this)• What are the performance implications?

10Implementing the R-B tree
• A new abstraction for the memory segment handles
uvm_physseg_t was introduced11Implementing the R-B tree
• A new abstraction for the memory segment handles
uvm_physseg_t was introduced• Utility functions, to ease the transition• Beforef o r ( l c v = 0 ; l c v < vm_nphysmem ; l c v ++) {seg = VM_PHYSMEM_PTR( l c v ) ;freepages += ( seg−>end − seg−>s t a r t ) ;}11

Implementing the R-B tree
• A new abstraction for the memory segment handles
uvm_physseg_t was introduced• Utility functions, to ease the transition• Beforef o r ( l c v = 0 ; l c v < vm_nphysmem ; l c v ++) {seg = VM_PHYSMEM_PTR( l c v ) ;freepages += ( seg−>end − seg−>s t a r t ) ;}• Afterf o r ( bank = u v m _ p h y s s e g _ g e t _ f i r s t ( ) ;uvm_physseg_valid ( bank ) ;bank = uvm_physseg_get_next ( bank ) ) {freepages += uvm_physseg_get_end ( bank ) −uvm_physseg_get_start ( bank ) ;}11

Implementing the R-B tree
• A new abstraction for the memory segment handles
uvm_physseg_t was introduced• Utility functions, to ease the transition• Beforef o r ( l c v = 0 ; l c v < vm_nphysmem ; l c v ++) {seg = VM_PHYSMEM_PTR( l c v ) ;freepages += ( seg−>end − seg−>s t a r t ) ;}• Afterf o r ( bank = u v m _ p h y s s e g _ g e t _ f i r s t ( ) ;uvm_physseg_valid ( bank ) ;bank = uvm_physseg_get_next ( bank ) ) {freepages += uvm_physseg_get_end ( bank ) −uvm_physseg_get_start ( bank ) ;}• An interesting utility function to note is
uvm_physseg_valid()
11

Testing uvm_physseg via ATF

Generic ATF Runs

• Baseline set of ATF tests written for the original static array
implementation

12

Generic ATF Runs

• Baseline set of ATF tests written for the original static array
implementation• rbtree(3) implementation would work as long as thebaseline ATF Tests passed.

12Generic ATF Runs

• Baseline set of ATF tests written for the original static array
implementation• rbtree(3) implementation would work as long as thebaseline ATF Tests passed.
• Overall this did reduce considerably the amount of time weneeded to spend to make sure the old and the new
implementation were working as expected12Generic ATF Runs

• Baseline set of ATF tests written for the original static array
implementation• rbtree(3) implementation would work as long as thebaseline ATF Tests passed.
• Overall this did reduce considerably the amount of time weneeded to spend to make sure the old and the new
implementation were working as expected• However, there were some interesting “Edge Cases”

12Case 1: uvm_page_physload()’s Prototype• Function was originally designed to plug in segments ofmemory range during boot time.
• If any errors happened it would generally print a message
and / or panic• It was fine for uvm_page_physload() to return void
after its execution in this scenario

13

Case 1: uvm_page_physload()’s Prototype• Function was originally designed to plug in segments ofmemory range during boot time.
• If any errors happened it would generally print a message
and / or panic• It was fine for uvm_page_physload() to return void
after its execution in this scenario• But this was NOT FINE for the ATF Testing13

Case 1: uvm_page_physload()’s Prototype

So what did we do?

14

Case 1: uvm_page_physload()’s Prototype

So what did we do?
We added a return value of type uvm_physmem_t

14Case 1: uvm_page_physload()’s Prototype

So what did we do?
We added a return value of type uvm_physmem_t
Old Prototype
void
uvm_page_physload ( paddr_t , paddr_t , paddr_t , paddr_t , i n t ) ;14

Case 1: uvm_page_physload()’s Prototype

So what did we do?
We added a return value of type uvm_physmem_t
Old Prototype
void
uvm_page_physload ( paddr_t , paddr_t , paddr_t , paddr_t , i n t ) ;New Prototype
uvm_physmem_t
uvm_page_physload ( paddr_t , paddr_t , paddr_t , paddr_t , i n t ) ;14

Case 1: uvm_page_physload()’s Prototype

So what did we do?
We added a return value of type uvm_physmem_t
Old Prototype
void
uvm_page_physload ( paddr_t , paddr_t , paddr_t , paddr_t , i n t ) ;New Prototype
uvm_physmem_t
uvm_page_physload ( paddr_t , paddr_t , paddr_t , paddr_t , i n t ) ;The tests became more concise, more readable and had
unwanted assumptions removed from within.
14

Case 2: Immutable handles• A particular test case uvm_physseg_get_prev kept
failing for static array implementation but not R-B Tree
implementation• For the static array implementation we were using the
VM_PSTRAT_BSEARCH strategy

15

Case 2: Immutable handles• A particular test case uvm_physseg_get_prev kept
failing for static array implementation but not R-B Tree
implementation• For the static array implementation we were using the
VM_PSTRAT_BSEARCH strategy• The test failed only if segments being inserted into thesystem out-of-order, this meant that the page frames ofthe segments that were inserted in chunks were not in asorted order

15

Case 2: Immutable handles• A particular test case uvm_physseg_get_prev kept
failing for static array implementation but not R-B Tree
implementation• For the static array implementation we were using the
VM_PSTRAT_BSEARCH strategy• The test failed only if segments being inserted into thesystem out-of-order, this meant that the page frames ofthe segments that were inserted in chunks were not in asorted order• Consequence of changing the way the handle of segment
was being referenced
15

Case 2: Immutable handles
Static array implementation+−−−−−+−−−−−+−−−−−+
| B |
|
|
+−−−−−−−−−−−−−−−−−+
Index| 0 | 1 | 2 |( uvm_physseg_t ) +−−−−−+−−−−−+−−−−−+

Segment I n f o+−−>+−−−−−+−−−−−+−−−−−+| A | B ||+−−−−−−−−−−−−−−−−−+| 0 | 1 | 2 |+−−−−−+−−−−−+−−−−−+

16

Case 2: Immutable handles
Static array implementation+−−−−−+−−−−−+−−−−−+
| B |
|
|
+−−−−−−−−−−−−−−−−−+
Index| 0 | 1 | 2 |( uvm_physseg_t ) +−−−−−+−−−−−+−−−−−+

Segment I n f o+−−>+−−−−−+−−−−−+−−−−−+| A | B ||+−−−−−−−−−−−−−−−−−+| 0 | 1 | 2 |+−−−−−+−−−−−+−−−−−+

R-B Tree implementation
+−−−+
| B |
+−−−+

+−−−+
|
|
+−−−+

+−−>

+−−−+
|
|
+−−−+

+−−−+
+−−−−+ B |
|
+−−−+
|
|
+−+−+
+−−−+
| A |
|
|
+−−−+
+−−−+

Note: The pointer to the nodes are the handles (uvm_physseg_t)16Case 2: Immutable handles• In order to separately identify this property of mutability weadded a new test case in ATF
uvm_physseg_handle_immutable

17

Case 2: Immutable handles• In order to separately identify this property of mutability weadded a new test case in ATF
uvm_physseg_handle_immutable• This test is expected to fail for static array implementation

17

Case 2: Immutable handles• In order to separately identify this property of mutability weadded a new test case in ATF
uvm_physseg_handle_immutable• This test is expected to fail for static array implementation• This test is expected to pass for R-B tree implementation

17

Case 2: Immutable handles• In order to separately identify this property of mutability weadded a new test case in ATF
uvm_physseg_handle_immutable• This test is expected to fail for static array implementation• This test is expected to pass for R-B tree implementation• This is important to notify the users of the old API and new
API about the potential pitfall of assuming the integrity of
the handle when writing new code.

17

Booting the KernelCase 1: The init dance
The first boot resulted in a kernel PANIC

18

Case 1: The init dance
The first boot resulted in a kernel PANIC• We quickly identified that kmem(9) is not available until
uvm_page_init() has done with all the initialization

18

Case 1: The init dance
The first boot resulted in a kernel PANIC• We quickly identified that kmem(9) is not available until
uvm_page_init() has done with all the initialization• Maintain a minimal “static array” whose size is
VM_PHYSSEG_MAX and once the init process is over,switch over to the kmem(9) allocator• uvm.page_init_done was used to distinguish when toswitch over to kmem(9)18Case 1: The init dance
The first boot resulted in a kernel PANIC• We quickly identified that kmem(9) is not available until
uvm_page_init() has done with all the initialization• Maintain a minimal “static array” whose size is
VM_PHYSSEG_MAX and once the init process is over,switch over to the kmem(9) allocator• uvm.page_init_done was used to distinguish when toswitch over to kmem(9)• We wrote wrappers for the kmem(9) allocators.
• uvm_physseg_alloc() and uvm_physseg_free()18

Case 1: The init dance
The first boot resulted in a kernel PANIC• We quickly identified that kmem(9) is not available until
uvm_page_init() has done with all the initialization• Maintain a minimal “static array” whose size is
VM_PHYSSEG_MAX and once the init process is over,switch over to the kmem(9) allocator• uvm.page_init_done was used to distinguish when toswitch over to kmem(9)• We wrote wrappers for the kmem(9) allocators.
• uvm_physseg_alloc() and uvm_physseg_free()• Wrote up the test cases for these first, allowing for asmooth implementation
18

Case 2: Fragmentation of segments
What exactly is “fragmentation of a segment”?

19Case 2: Fragmentation of segments
What exactly is “fragmentation of a segment”?
The pgs[] is contained in a given segment, allocated by
kmem(9) allocators+−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−+|
Segment A|+−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−+

19

Case 2: Fragmentation of segments
What exactly is “fragmentation of a segment”?
The pgs[] is contained in a given segment, allocated by
kmem(9) allocators+−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−+|
Segment A|+−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−+

So what happens to pgs[] if we “unplug” a section?
+−−−−−−−−−−−−−−−−−−−−−−−−−−−+−−−−−−−−−−−−−−−−+|
Segment A|
Segment B|+−−−−−−−−−−−−−−−−−−−−−−−−−−−+−−−−−−−−−−−−−−−−+

19

Case 2: Fragmentation of segments
What exactly is “fragmentation of a segment”?
The pgs[] is contained in a given segment, allocated by
kmem(9) allocators+−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−+|
Segment A|+−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−+

So what happens to pgs[] if we “unplug” a section?
+−−−−−−−−−−−−−−−−−−−−−−−−−−−+−−−−−−−−−−−−−−−−+|
Segment A|
Segment B|+−−−−−−−−−−−−−−−−−−−−−−−−−−−+−−−−−−−−−−−−−−−−+

What happens to pgs[] if we “unplug” from the middle?
+−−−−−−−−−−−−−−+−−−−−−−−−−−−−−+−−−−−−−−−−−−−−+|
Segment A |Segment C | Segment B|+−−−−−−−−−−−−−−+−−−−−−−−−−−−−−+−−−−−−−−−−−−−−+
19

Case 2: Fragmentation of segments

How did we solve this?

20

Case 2: Fragmentation of segments

How did we solve this?
• Use the extent(9) memory manager to manage thepgs[] array

20

Case 2: Fragmentation of segments

How did we solve this?
• Use the extent(9) memory manager to manage thepgs[] array• We applied the “init dance” technique to solve Boot time vsnon-Boot time allocation of slabs

20

Case 2: Fragmentation of segments

How did we solve this?
• Use the extent(9) memory manager to manage thepgs[] array• We applied the “init dance” technique to solve Boot time vsnon-Boot time allocation of slabs• Once again extensive ATF tests that helped us out inminimising the downtime from debugging the code20

Performance evaluation

Designing the test framework...so we leveraged ATF to do this
• The most frequent opeation is uvm_physseg_find()21

Designing the test framework...so we leveraged ATF to do this
• The most frequent opeation is uvm_physseg_find()• Copied over the PHYS_TO_VM_PAGE() macro and therelated code from uvm_page.c

21Designing the test framework...so we leveraged ATF to do this
• The most frequent opeation is uvm_physseg_find()• Copied over the PHYS_TO_VM_PAGE() macro and therelated code from uvm_page.c• Plug in segments and then do multiple calls to
PHYS_TO_VM_PAGE()f o r ( i n t i = 0 ; i < 100; i ++) {pa = ( p a d d r _ t ) random ( ) % ( a d d r _ t ) c t o b ( VALID_END_PFN_1 ) ;PHYS_TO_VM_PAGE( pa ) ;}21

Designing the test framework...so we leveraged ATF to do this
• The most frequent opeation is uvm_physseg_find()• Copied over the PHYS_TO_VM_PAGE() macro and therelated code from uvm_page.c• Plug in segments and then do multiple calls to
PHYS_TO_VM_PAGE()f o r ( i n t i = 0 ; i < 100; i ++) {pa = ( p a d d r _ t ) random ( ) % ( a d d r _ t ) c t o b ( VALID_END_PFN_1 ) ;PHYS_TO_VM_PAGE( pa ) ;}• After some tweaking around we managed to write up thetests varying from 100 calls to 100 Million calls
21Designing the test framework

Things to Note
• This methodology is not a perfect load test since there is acall to random()• This will cumulatively add up to the runtime of the function
we are trying to load test.

22

Designing the test framework

Things to Note
• This methodology is not a perfect load test since there is acall to random()• This will cumulatively add up to the runtime of the function
we are trying to load test.
• All of the ATF tests have ATF_CHECK_EQ(true, true)at the bottom of the test indicating the test will never fail• This is done because the test is NOT a check ofcorrectness

22

Designing the test framework
We implemented two types of test strategies• Fixed size segment: Here we plug in a “fixed” size
segment. And pick a random address to do the
PHYS_TO_VM_PAGE(). The variable here was the amountof calls done to PHYS_TO_VM_PAGE()

23

Designing the test framework
We implemented two types of test strategies• Fixed size segment: Here we plug in a “fixed” size
segment. And pick a random address to do the
PHYS_TO_VM_PAGE(). The variable here was the amountof calls done to PHYS_TO_VM_PAGE()• Fragmented segment: Here we plug in a known size
segment. After which we start unplugging areas of thememory. Then we pick a random address to do
PHYS_TO_VM_PAGE(). Here the variable was the memory
size meaning, the bigger memory segment the more
fragmented it was.
23

Designing the test framework
An example run of these tests with the standard atf-run
piped through atf-report will have a similar output.
Note: In the results 100 consecutive runs were done and then the average, minimum
and maximum runtimes were calculated.
t_uvm_physseg_load ( 1 / 1 ) : 11 t e s t cases
uvm_physseg_100 : [0.003286 s ] Passed .
uvm_physseg_100K : [0.010982 s ] Passed .
uvm_physseg_100M : [8.842482 s ] Passed .
uvm_physseg_10K : [0.004398 s ] Passed .
uvm_physseg_10M : [0.954270 s ] Passed .
uvm_physseg_128MB : [2.176629 s ] Passed .
uvm_physseg_1K : [0.002702 s ] Passed .
uvm_physseg_1M : [0.094821 s ] Passed .
uvm_physseg_1MB : [0.984185 s ] Passed .
uvm_physseg_256MB : [2.485398 s ] Passed .
uvm_physseg_64MB : [0.914363 s ] Passed .
[16.478686 s ]Summary f o r 1 t e s t programs :11 passed t e s t cases .
0 f a i l e d t e s t cases .
0 expected f a i l e d t e s t cases .
0 skipped t e s t cases .

24

Benchmark results

Calls to PHYS_TO_VM_PAGE()
Test Name
uvm_physseg_100
uvm_physseg_1K
uvm_physseg_10K
uvm_physseg_100K
uvm_physseg_1M
uvm_physseg_10M
uvm_physseg_100M

Average
0.004599
0.002740
0.003491
0.011424
0.093359
0.892827
8.932540

Minimum
0.003286
0.001991
0.002836
0.009388
0.079128
0.813503
8.434525

Maximum
0.010213
0.005747
0.007941
0.017161
0.138379
1.172205
11.616543

Table 1: R-B tree implementation

Test Name
uvm_physseg_100
uvm_physseg_1K
uvm_physseg_10K
uvm_physseg_100K
uvm_physseg_1M
uvm_physseg_10M
uvm_physseg_100M

Average
0.004714
0.002754
0.003585
0.011007
0.086208
0.843048
8.434760

Minimum
0.003511
0.002088
0.002666
0.009199
0.076989
0.782676
8.128623

Maximum
0.013895
0.005318
0.005271
0.016627
0.116637
0.980598
9.132065

25

Table 2: Static array implementation

Calls to PHYS_TO_VM_PAGE()

Figure 1: A closer look at the 10M and 100M calls side-by-side26

Calls to PHYS_TO_VM_PAGE()
Since the 100M calls, took the most amount of time, we did somevery specific analysis on this.
We calculated the Average, Standard Deviation (Population) andMargin of Error with a 95% confidence interval.
In a total of 100 runs, the random() function contributed to roughly 2.03 seconds forthe average runtime, for a 100 Million calls to PHYS_TO_VM_PAGE().

AverageStandard Deviation
Margin of Error

Static Array
8.43476
0.19331±0.03789

R-B Tree8.932540.41553±0.08144

Table 3: Comparison of the average, standard deviation and marginof error for the 100M calls to PHYS_TO_VM_PAGE()
27

Calls to PHYS_TO_VM_PAGE()

Figure 2: Clearly there is a 5.59% degradation in performance with
the R-B tree implementation

28

Calls to PHYS_TO_VM_PAGE() after fragmentation• Number after test name indicates the amount of memoryon which fragmentation was done
• Fragmentation was done by uvm_physseg_unplug()• After unplug was completed PHYS_TO_VM_PAGE() was
called 10M (million) times for every test.
Test Name
uvm_physseg_1MB
uvm_physseg_64MB
uvm_physseg_128MB
uvm_physseg_256MB

Average
1.015810
0.958675
2.1552702.550920

Minimum
0.941942
0.877151
2.024838
2.360252

Maximum
1.361913
1.279663
2.866540
3.736369

Table 4: Comparison of average, minimum and maximum execution
times of various load tests with uvm_hotplug(9) enabled on
fragmented memory segments.
29

Calls to PHYS_TO_VM_PAGE() after fragmentation

Figure 3: R-B tree performance for 10M Calls to
PHYS_TO_VM_PAGE() after fragmentation at every 8 PFN

30

Conclusion and future work

Retrospective

Looking back...
• rumpkernel(7) based testing?
• Code coverage, maybe?
• Performance testing in an actual live kernel implementation
with dtrace(1)31Conclusion• Systems Programming can be made much less stressful
by using existing Software Engineering techniques.

32

Conclusion• Systems Programming can be made much less stressful
by using existing Software Engineering techniques.
• The availability of general purpose APIs such as
rbtree(3) and extent(9) in the NetBSD kernel, whichmakes implementation much less headache.

32

Future work

• We would like to encourage other NetBSD developers to
use this API to write hotplug/ unplug drivers for theirfavourite platforms with suitable hardware.

33

Future work

• We would like to encourage other NetBSD developers to
use this API to write hotplug/ unplug drivers for theirfavourite platforms with suitable hardware.
• We also encourage other BSDs to pick up our work - since
this will clean up the current legacy implementations which
are pretty much identical.

33

Credits and References

Thank you• The NetBSD Foundation <http://www.NetBSD.org/foundation> generously
funded this work.
• KeK <hello@kek.org.in> provided a cozy space right next to Kovalam Beach for
us to hammer out the implementation.
• Chuck Silvers <chs@NetBSD.org> reviewed and helped refine the APIs. He
also provided deep insight into the challenges of architecting such low level code.
• Matthew Green <mrg@NetBSD.org> made many helpful suggestions andcritical feedback during the development and integration timeframe.
• Maya Rashish <maya@NetBSD.org> helped expose the API to multiple
usecase situations (including header breakage in pkgsrc).
• Nick Hudson <skrll@NetBSD.org> contributed bugfixes, testing and integration
on a wide range of hardware ports.
• Philip Paeps <philip@FreeBSD.org> helped guide creation, review andcorrection of the content of abstract and paper for uvm_hotplug(9)• Thomas Klausner <wiz@NetBSD.org> helped make corrections to man page of
uvm_hotplug(9)• Tom Flavel <tom@printf.net> coerced cherry@NetBSD.org towards TDD, who
was able to interest Santhosh Raju in applying the method to kernel
programming. This allegedly turned out to be a good thing eventually.
34

Thank you... And to all others who helped us along the way and we may
have accidentally missed out or forgot to mention.

35

Thank you... And to all others who helped us along the way and we may
have accidentally missed out or forgot to mention.
And of course the audience for being here and patient while
listening to the talk.

35

Questions

36

References• uvm_hotplug(9) man page
http://netbsd.gw.com/cgi-bin/man-cgi?uvm_hotplug++NetBSD-current• uvm_hotplug(9) port-masters’ FAQ
https://wiki.netbsd.org/features/uvm_hotplug/• uvm(9) man page
http://netbsd.gw.com/cgi-bin/man-cgi?uvm+9+NetBSD-current• rbtree(3) man page
http://netbsd.gw.com/cgi-bin/man-cgi?rbtree+3+NetBSD-current• atf(7) man page
http://netbsd.gw.com/cgi-bin/man-cgi?atf+7+NetBSD-current• uvm_hotplug(9) development blog
http://fraggerfox.homenet.org:10080/bsd-blog/

37

The End