nonlinear: disable MCTRACE2 for a massive speedup
time goes from 90 minuttes to 2.5 minutes (at home).
Maybe transferring MCTRACE2 back and forth across the C call is expensive?
try sqrt() of weights. epsilon [.001, .004]
Without adjusting epsilo sample size is 0.
After divide by 10
> effectiveSize(MCTRACE[,1:15])
  intrcpt       age   rac_gay nPartners pEthnic.1 pEthnic.2 pEthnic.3 pEthnic.4
15.680617 38.964382 57.662331 69.996937 16.554433 31.579550 12.454586 18.248125
 ethnic.1  ethnic.2 HIVTest.1 HIVTest.2       tau    theta1    theta2
17.109249 38.734289 20.068314 14.139029 13.484799 12.251103  2.693618

> summary(effectiveSize(MCTRACE))
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
 0.8663  2.9370  4.3360  5.1620  6.3200 70.0000

previous effective sizes where mostly better, though my main goal is to reduce
variability, which is a closer call.
> effectiveSize(MCTRACE[,1:15])
      var1       var2       var3       var4       var5       var6       var7
 39.259942  32.230820  40.897515  81.661921  40.250163  84.681450 213.774218
      var8       var9      var10      var11      var12      var13      var14
 39.599375  27.670555  58.960078  71.066753  27.024409   1.937492  71.454085
     var15
  2.548817
double epsilon with sqrt() [.002, .008]
little difference, but notice the drop for nPartners
> summary(effectiveSize(MCTRACE))
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
 0.8929  3.2810  4.6920  5.5270  6.9420 45.1200
> effectiveSize(MCTRACE[,1:15])
  intrcpt       age   rac_gay nPartners pEthnic.1 pEthnic.2 pEthnic.3 pEthnic.4
10.791441 20.599277 34.434800  9.462951 24.126532 45.121524 22.501963 20.580983
 ethnic.1  ethnic.2 HIVTest.1 HIVTest.2       tau    theta1    theta2
16.521965 36.613397 26.186730 17.515471 13.328343  7.163809  5.485239
double step size again to [.004, .016]
effective sizes are worse, though it's not clear the range of variation is.
> effectiveSize(MCTRACE[,1:15])
  intrcpt       age   rac_gay nPartners pEthnic.1 pEthnic.2 pEthnic.3 pEthnic.4
 3.909459 19.415757 29.147337 31.450544  7.960255  7.732431  4.246762  4.043062
 ethnic.1  ethnic.2 HIVTest.1 HIVTest.2       tau    theta1    theta2
11.761150 13.995851 16.659460  5.694716 27.435516  1.036956  3.289481
> summary(effectiveSize(MCTRACE))
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
 0.7591  2.2590  3.4150  4.0790  5.0170 31.4500
step back to  [.001, .004], double # steps to 40. better
run time 284s; was 150s with 20 steps
sample size is mostly better, variation is perhaps better.
but note tau is very weak
> summary(effectiveSize(MCTRACE))
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
 0.9787  5.7820  8.2920  9.0740 10.9600 66.2000
> effectiveSize(MCTRACE[,1:15])
  intrcpt       age   rac_gay nPartners pEthnic.1 pEthnic.2 pEthnic.3 pEthnic.4
23.145767 21.272095 54.892581 29.463440 57.844134 61.483626 35.709460 39.791021
 ethnic.1  ethnic.2 HIVTest.1 HIVTest.2       tau    theta1    theta2
35.800903 66.197990 15.595368 22.367149  1.967924 12.294117  2.744720
back to 1/() mass.
effective sizes are worse but ratio of them (Q3/Q1) is a bit better.
Note I was using a bigger step size before.
> summary(effectiveSize(MCTRACE))
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
 0.7944  4.2190  6.4820  7.2860  9.1330 52.2900
> effectiveSize(MCTRACE[,1:15])
  intrcpt       age   rac_gay nPartners pEthnic.1 pEthnic.2 pEthnic.3 pEthnic.4
15.636455 23.167538 17.424620 52.286588 15.175572 26.628120 23.576350  8.929568
 ethnic.1  ethnic.2 HIVTest.1 HIVTest.2       tau    theta1    theta2
14.435302 46.368148 14.776402  7.155068  3.520129  9.189152  3.240913
step size x 5 much better [.005, .020]
except tau is worse
> summary(effectiveSize(MCTRACE))
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
  1.123  36.270  73.680  81.040 108.800 563.600
> effectiveSize(MCTRACE[,1:15])
   intrcpt        age    rac_gay  nPartners  pEthnic.1  pEthnic.2  pEthnic.3
 56.984292  45.450854  91.826550  51.549007 108.838396 129.135245 149.762914
 pEthnic.4   ethnic.1   ethnic.2  HIVTest.1  HIVTest.2        tau     theta1
 63.996579  43.950478  80.364775  89.434732  42.226728   2.359491 152.846523
    theta2
  6.128133
step x 5 [.025, .100] worse
> summary(effectiveSize(MCTRACE))
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
  1.211   9.613  16.590  17.680  23.110  93.160
> effectiveSize(MCTRACE[,1:15])
  intrcpt       age   rac_gay nPartners pEthnic.1 pEthnic.2 pEthnic.3 pEthnic.4
18.584769 17.453713 24.766913 29.692833 16.440683 17.375495 19.930532 13.443269
 ethnic.1  ethnic.2 HIVTest.1 HIVTest.2       tau    theta1    theta2
19.656174 24.480126 15.029838 22.722851  4.580420 26.128476  7.872494
back to initial [.01, .04] step size. but 40 steps. best so far
However q3/q1 is bigger, and tau still stinks
> summary(effectiveSize(MCTRACE))
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
  2.187  52.740 118.000 127.300 174.500 720.600
> effectiveSize(MCTRACE[,1:15])
   intrcpt        age    rac_gay  nPartners  pEthnic.1  pEthnic.2  pEthnic.3
 95.800388  79.221080  83.514567 115.017099 204.255764 164.943329 268.354061
 pEthnic.4   ethnic.1   ethnic.2  HIVTest.1  HIVTest.2        tau     theta1
133.022785  42.766035  80.256574  81.134976  30.330322   3.741292 249.224270
    theta2
 13.725845
80 steps: better still
546s at home, a bit under twice the time, but effective sample size
quadruples
even tau has signs of life
[1] "Thu Jul 11 10:52:26 2013"
> colnames(MCTRACE) <- c(mynames, "tau",
+                        paste("theta", seq(1061), sep=""))
> summary(effectiveSize(MCTRACE))
    Min.  1st Qu.   Median     Mean  3rd Qu.     Max.
   3.666  197.400  464.800  465.200  670.400 1510.000
> effectiveSize(MCTRACE[,1:15])
  intrcpt       age   rac_gay nPartners pEthnic.1 pEthnic.2 pEthnic.3 pEthnic.4
222.92664 301.21375 122.61388 259.47370 335.80081 359.38386 294.74163 391.46848
 ethnic.1  ethnic.2 HIVTest.1 HIVTest.2       tau    theta1    theta2
262.36012 299.04668  86.51835  73.13142  21.19068 837.40905  42.31302
steps x 2 to 160, iterations /2 to 500
539s not quite as strong in general, though Q1 is better.
> summary(effectiveSize(MCTRACE))
    Min.  1st Qu.   Median     Mean  3rd Qu.     Max.
   3.592  255.400  382.700  378.300  500.000 1390.000
> effectiveSize(MCTRACE[,1:15])
  intrcpt       age   rac_gay nPartners pEthnic.1 pEthnic.2 pEthnic.3 pEthnic.4
282.80162 284.43541 220.01144 395.83836 203.46361 376.57326 279.10166 387.61282
 ethnic.1  ethnic.2 HIVTest.1 HIVTest.2       tau    theta1    theta2
249.47060 202.81904 207.67153 186.96654  20.10318 387.59644  84.41377
tau stepsize hacked x 10.  a bit worse
538s
> summary(effectiveSize(MCTRACE))
    Min.  1st Qu.   Median     Mean  3rd Qu.     Max.
   1.749  235.200  345.700  336.500  429.500 1065.000
> effectiveSize(MCTRACE[,1:15])
  intrcpt       age   rac_gay nPartners pEthnic.1 pEthnic.2 pEthnic.3 pEthnic.4
242.68244 414.03520 206.58462 274.13213 278.55947 268.67265 283.20258 292.51742
 ethnic.1  ethnic.2 HIVTest.1 HIVTest.2       tau    theta1    theta2
212.24142 186.27014 209.03936 159.31734  21.05638 327.10818  82.49950
tau stepsize now 4x original, 40% last commit.
a bit better than x10, mostly not quite as good as original,
though tau effective size 26.4 here vs 20.1 original
> summary(effectiveSize(MCTRACE))
    Min.  1st Qu.   Median     Mean  3rd Qu.     Max.
   1.819  247.400  368.600  364.200  500.000 1183.000
> effectiveSize(MCTRACE[,1:15])
  intrcpt       age   rac_gay nPartners pEthnic.1 pEthnic.2 pEthnic.3 pEthnic.4
199.83648 228.17296 199.83818 230.06223 223.13229 335.02771 159.17613 292.99009
 ethnic.1  ethnic.2 HIVTest.1 HIVTest.2       tau    theta1    theta2
183.91398 220.61610 269.03997 183.87833  26.44597 350.02166  70.83440
try weighting by 1/sqrt(2nd deriv).  blows up
dLogDens is evaluated with parameters
summary(q)
       Min.     1st Qu.      Median        Mean     3rd Qu.        Max.
 -1.000e+00   1.338e+04  2.647e+162         Inf  6.663e+162         Inf
though there is no error message.

Further up the stack, weights were set to
summary(weights)
     Min.   1st Qu.    Median      Mean   3rd Qu.      Max.
0.0005832 0.4122000 0.4677000 0.4477000 0.5166000 0.5166000
divide step size by 10.  doesn't help; still blows up.
Error in term1[iRegular] <- dens[iRegular]/cum[iRegular] (from mice.impute.2l.nonlinear.r#291) :
  NAs are not allowed in subscripted assignments
For reference, here are the different weightings
newton, which I was using an will revert to, was d/d2
d = first deriv
d2 = 2nd deriv (numeric)
Browse[2]> data.frame(d=danalytic, d2=d2, newton=d2a, sd=1/sqrt(abs(d2)))[1:15,]
               d            d2        newton           sd
1    -88.0825458 -2.045788e+03  0.0430555557 0.0221090291
2  -3318.7867694 -2.876249e+06  0.0011538594 0.0005896398
3    -70.5543714 -1.260854e+04  0.0055957621 0.0089056917
4   2528.4725814 -2.939897e+06 -0.0008600549 0.0005832221
5     13.6170696 -2.223315e+02 -0.0612467001 0.0670655544
6    -32.7263035 -7.341088e+02  0.0445796353 0.0369079377
7    -13.8127084 -9.874728e+01  0.1398793747 0.1006323049
8     -3.7501604 -4.405925e+02  0.0085116298 0.0476410625
9     32.1692453 -6.009283e+02 -0.0535325884 0.0407932856
10   -68.0796605 -7.468578e+02  0.0911547849 0.0365915701
11     8.3995933 -2.914872e+02 -0.0288163321 0.0585720227
12    -6.2769834 -8.035262e+02  0.0078117964 0.0352776762
13   -10.7417993 -6.928593e+03  0.0015503580 0.0120137193
14    -2.3240851 -3.746448e+00  0.6203436676 0.5166425369
15    -0.1067854 -6.433835e+00  0.0165974695 0.3942439589
back to the weighting scheme and step I've been using the last few days.
80 steps, only 500 iterations
277s
> summary(effectiveSize(MCTRACE))
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
   1.51  108.60  248.70  260.70  369.70 1183.00
> effectiveSize(MCTRACE[,1:15])
  intrcpt       age   rac_gay nPartners pEthnic.1 pEthnic.2 pEthnic.3 pEthnic.4
171.84853 240.93449  45.13615 145.42384 165.93196 245.30682 279.96554 196.95002
 ethnic.1  ethnic.2 HIVTest.1 HIVTest.2       tau    theta1    theta2
137.04266 152.65984  90.39987  19.87017  16.41387 382.40064  30.71067
switch to Albert + Chibb for timing comparisons. 500 iterations.
223s @ home
disabled all tracing in it.
Unfortunately, this leaves no way to see intermediate results.
reenable tracing
223s at home; no slowdown.  distinctly worse than hybrid MC, which is only
a bit slower.  However, some of the effectiveSizes here are superior (HIVTest, theta2).
> summary(effectiveSize(MCTRACE[2:501,1:1074]))
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
  11.05  104.10  168.50  169.40  235.20  393.70
> effectiveSize(MCTRACE[2:501,1:15])
       tau   intrcpt0       age0   rac_gay0 nPartners0 pEthnic.10 pEthnic.20
  11.04973   34.67292   44.71204   29.71229   16.94736   65.41444   57.42786
pEthnic.30 pEthnic.40  ethnic.10  ethnic.20 HIVTest.10 HIVTest.20     theta1
  57.73787   67.38252   70.29024   50.49924   31.09470   47.16445  266.27641
    theta2
 244.15685
back to using hybrid MC
inital tau x 2.5. -> 0 effective sample size
Since our estimates are low relative to glmer and to where tau seems to
be heading.