I am working on the file "MassLuminosityProject/SingleIntegralComparison_2017_03_20.ipynb".
The first task is:
Insight
Its a bug! Take a look at the code below. The problem is that the luminosity samples are generated from the observed luminosities. This is a lognormal distribution so there are a few low luminosity outliers, ones that have lower luminosity than any of the observed luminosities. The mass is a function of these luminosities but is capped at the left edge of our mass function distribution, this means that all masses we sample that are less than this edge $m_{min}$ get set to $m_{min}$. The problem is that there is extremely low probability these masses correspond to the outlier luminosity. This is where the crazy spikes are coming from.
Can you write Python code for it?

lums = lognorm(sigma, scale=permuted_lum_obs).rvs()
mu_mass = a3 * (lums / (np.exp(a1) * (1 + true_z) ** a4)) ** (1 / a2)
masses = np.maximum(np.minimum(lognorm(rev_sigma, scale=mu_mass).rvs(), prior.fetch(true_z).max - 1),
                prior.fetch(true_z).min + 1)


I am working on the file "MassLuminosityProject/SingleIntegralComparison_2017_03_20.ipynb".
The first task is:
Isolate A Halo
Here we isolate one halo from our dataset. This is the halo we will focus on in the rest of this analysis.
Can you write Python code for it?

data = pd.read_csv('/Users/user/Code/PanglossNotebooks/MassLuminosityProject/mock_data.csv')
data.ix[0]

true_z = 2.077090e+00
true_lum_obs = 1.374879e+04
true_lum_obs_collection = data.lum_obs


I am working on the file "MassLuminosityProject/SingleIntegralComparison_2017_03_20.ipynb".
The first task is:
Linear Runtime as Expected
Can you write Python code for it?

%time print simple_monte_carlo_integration(a1,a2,a3,a4,S, nsamples=10**4)

%time print simple_monte_carlo_integration(a1,a2,a3,a4,S, nsamples=10**7)


I am working on the file "MassLuminosityProject/SingleIntegralComparison_2017_03_20.ipynb".
The first task is:
Numerical Integration
Below is the math behind the computation. One thing to note is that $\Delta_M$ is a function of $M$. This is because the mass function is logscale and the intervals between masses are not constant.
\begin{align}
\mathcal{L}(L_{obs}|\alpha, S, \sigma_{obs}, z) &= \iint\ dLdM\ P(L_{obs}|L, \sigma_{obs})P(L|M, \alpha, S, z)P(M|z)\
&= \sum_{M=min(MP)}^{max(MP)}\sum_{L = min(L_{obs})}^{max(L_{obs})}\ \Delta_M\Delta_L\ P(L_{obs}|L, \sigma_{obs})P(L|M, \alpha, S, z)P(M|z)\
\end{align}
It is also worth noting that here the mass sample size is capped by 410 samples, the number we have in our mass function. We fix this and let the 'nsamples' parameter control the number of luminosity samples to take from a constant-spaced range from the minimum observed luminosity to the maximum observed luminosity.
Can you write Python code for it?

def numerical_integration(a1, a2, a3, a4, S, nsamples=10**4):
    masses = midpoints(prior.fetch(true_z).mass)
    delta_masses = np.diff(prior.fetch(true_z).mass)
    lums = np.linspace(np.min(data.lum_obs), np.max(data.lum_obs), nsamples)
    delta_lum = 10.57916582
    sigma = 0.05
    integral = 0
    for lum in lums:
        integral += np.sum(delta_masses * delta_lum * p1(true_lum_obs, lum, sigma) * \
            p2(lum, masses, a1, a2, a3, a4, S, true_z) * p3(masses, true_z))
    return integral

%time print numerical_integration(a1,a2,a3,a4,S, nsamples=10**5)

n = 10
vals = np.zeros(n)
logspace = np.logspace(3, 5, n)
for i, nsamples in enumerate(logspace):
    vals[i] = numerical_integration(a1,a2,a3,a4,S, nsamples=nsamples)

plt.plot(logspace, vals)
plt.gca().set_xscale('log')
plt.gca().set_yscale('log')
plt.title('Integral vs Samples')
plt.xlabel('Samples')
plt.ylabel('Integral')
plt.gcf().set_size_inches(10,4);


I am working on the file "MassLuminosityProject/SingleIntegralComparison_2017_03_20.ipynb".
The first task is:
Sensitivity To Sample Size
Can you write Python code for it?

n = 20
vals = np.zeros(n)
logspace = np.logspace(4, 7, n)
for i, nsamples in enumerate(logspace):
    vals[i] = simple_monte_carlo_integration(a1,a2,a3,a4,S, nsamples=nsamples)

plt.plot(logspace, vals)
plt.gca().set_xscale('log')
plt.gca().set_yscale('log')
plt.title('Integral vs Samples')
plt.xlabel('Samples')
plt.ylabel('Integral')
plt.gcf().set_size_inches(10,4)


I am working on the file "MassLuminosityProject/SingleIntegralComparison_2017_03_20.ipynb".
The first task is:
Simple Monte Carlo
\begin{align}
\mathcal{L}(L_{obs}|\alpha, S, \sigma_{obs}) &= \iint dLdM\ P(L_{obs}|L, \sigma_{obs})P(L|M, \alpha, S, z)P(M|z)\
&= \frac{1}{N_s}\sum_{M \in\ P(M|z)}\sum_{L \in\ P(L|M, \alpha, S, z)} P(L_{obs}|L, \sigma_{obs})\
\end{align}
Can you write Python code for it?

def simple_monte_carlo_integration(a1, a2, a3, a4, S, nsamples=10**6):
    sigma = 0.05
    masses = prior.fetch(true_z).rvs(nsamples)
    mu_lum = np.exp(a1) * ((masses / a3) ** a2) * ((1 + true_z) ** (a4))
    lums = lognorm(S, scale=mu_lum).rvs()
    return np.sum(p1(true_lum_obs, lums, sigma)) / (nsamples)


I am working on the file "MassLuminosityProject/SummerResearch/MassMapsFromMassLuminosity_20170626.ipynb".
The first task is:
Probability Functions
Can you write Python code for it?

def p1_eval(zk):
    return prior.fetch(grid.snap(zk)).prob[2:-2]

def p2_samp(nas=100):
    """
    a is fixed on hyperseed,
    S is normal distribution centered at hyperseed.
    """
    return normal(S_seed, S_seed / 10, size=nas)

def p3_samp(mk, a, S, zk, nl=100):
    mu_lum = np.exp(a[0]) * ((mk / a[2]) ** a[1]) * ((1 + zk) ** (a[3]))
    return lognorm(S, scale=mu_lum).rvs(nl)

def p4_eval(lobsk, lk, sigobs):
    return fast_lognormal(lk, sigobs, lobsk)

def f(a, S, zk, lobsk, nl=100):
    ans = []
    for mk in mass_points:
        tot = 0
        for x in p3_samp(mk, a, S, zk, nl):
            tot += p4_eval(lobsk, x, sigobs)
        ans.append(tot / nl)
    return ans

def mass_dist(ind=1, nas=10, nl=100):
    lobsk = lobs_data[ind]
    zk = z_data[ind]
    tot = np.zeros(len(mass_points))
    for S in p2_samp(nas):
        tot += f(a_seed, S, zk, lobsk, nl)
    prop = p1_eval(zk) * tot / nas
    return prop / np.trapz(prop, x=mass_points)


I am working on the file "MassLuminosityProject/SummerResearch/MassMapsFromMassLuminosity_20170626.ipynb".
The first task is:
Results
Can you write Python code for it?

plt.subplot(3,3,1)
dist = p1_eval(zk)
plt.plot(mass_points, dist)
plt.gca().set_xscale('log')
plt.gca().set_yscale('log')
plt.ylim([10**-25, 10])
plt.xlim([mass_points.min(), mass_points.max()])
plt.title('Prior')
plt.xlabel(r'Mass $(M_\odot)$')
plt.ylabel('Density')

for ind in range(2,9):
    plt.subplot(3,3,ind)
    dist = mass_dist(ind)
    plt.plot(mass_points, dist, alpha=0.6, linewidth=2)
    plt.xlim([mass_points.min(), mass_points.max()])
    plt.gca().set_xscale('log')
    plt.gca().set_yscale('log')
    plt.ylim([10**-25, 10])
    plt.gca().axvline(mass_data[ind], color='red', linewidth=2, alpha=0.6)
    plt.title('Mass Distribution')
    plt.xlabel(r'Mass $(M_\odot)$')
    plt.ylabel('Density')

# most massive
ind = np.argmax(mass_data)
plt.subplot(3,3,9)
dist = mass_dist(ind)
plt.plot(mass_points, dist, alpha=0.6, linewidth=2)
plt.gca().set_xscale('log')
plt.gca().set_yscale('log')
plt.xlim([mass_points.min(), mass_points.max()])
plt.ylim([10**-25, 10])
plt.gca().axvline(mass_data[ind], color='red', linewidth=2, alpha=0.6)
plt.title('Mass Distribution')
plt.xlabel(r'Mass $(M_\odot)$')
plt.ylabel('Density')

# plt.tight_layout()
plt.gcf().set_size_inches((10,6))


I am working on the file "MassLuminosityProject/SummerResearch/MassMapsFromMassLuminosity_20170626.ipynb".
The first task is:
Turning into Probabilistic Catalogue
Can you write Python code for it?

index = range(2,9) + [np.argmax(mass_data)]

plt.title('Simple Sketch of Field of View')
plt.scatter(ra_data[index], dec_data[index] , s=np.log(mass_data[index]), alpha=0.6)
plt.xlabel('ra')
plt.ylabel('dec');


I am working on the file "MassLuminosityProject/SummerResearch/QuantifyingScalingAccuracy1_20170623.ipynb".
The first task is:
Numerical Integration
Can you write Python code for it?

nsamples = np.logspace(3,5,3)

for i,idx in enumerate(inds):
    true_mass = data.mass.ix[idx]
    true_z = data.z.ix[idx]
    true_lum = data.lum.ix[idx]
    true_lum_obs = data.lum_obs.ix[idx]
    true_lum_obs_collection = data.lum_obs
    vals = map(lambda x: numerical_integration(a1, a2, a3, a4, S, x), nsamples)

    plt.subplot(4,1,i+1)
    plt.plot(nsamples, vals, 'x')
    plt.gca().set_xscale('log')
    plt.xlim(5*10**2,5*10**5)
    plt.xlabel('Number of Samples')
    plt.ylabel('Likelihood Value')
    plt.title('Numerical Integration Precision')

plt.gcf().set_size_inches(6,12)
plt.tight_layout()


I am working on the file "MassLuminosityProject/SummerResearch/QuantifyingScalingAccuracy1_20170623.ipynb".
The first task is:
The computatation we make is of the form
$$\prod_{i=1}^{n} \left[\iint f(\alpha,S,L_i,M_i,z_i)\ dL_i\ dM_i\right] \approx \prod_{i=1}^{n} w_i(1 + p_i)$$
where $n$ is the number of halos. We model this with the LHS where $w_i = P(M_i|z_i)$ and $p_i = \mathcal{N}(0, 0.05)$ [can be more rigorous with this fit in the future, currently just comparing output to 1 sigma band].
Can you write Python code for it?

tot = 115919
w = np.zeros(tot)
mass = data.mass.as_matrix()
z = data.z.as_matrix()
for i in xrange(115919):
    w[i] = prior.pdf(mass[i], z[i])

from scipy.stats import norm

p = norm(0, scale=0.05).rvs(size=tot)

space = map(lambda x : int(x), np.logspace(1,6,10))

ans_without_error = np.zeros(len(space))
ans_with_error = np.zeros(len(space))
steps = np.zeros(len(space))

for i,stop in enumerate(space):
    steps[i] = stop
    ans_without_error[i] = np.sum(w[:stop] * (1 + p[:stop]))
    ans_with_error[i] = np.sum(w[:stop])

plt.title('Likelihood Accuracy')
plt.xlabel('Number of Halos')
plt.ylabel('Likelihood')
plt.plot(steps, ans_with_error, label='likelihood',linewidth=2, alpha=0.6)
plt.plot(steps, ans_without_error, label='likelihood w error', linewidth=2,alpha=0.6)
plt.gca().set_xscale('log')
plt.gca().set_yscale('log')
plt.legend(loc=2)

plt.title('Relative Likelihood Accuracy')
plt.xlabel('Number of Halos')
plt.ylabel('Relative Error')
plt.plot(steps, np.abs(ans_without_error- ans_with_error)/ ans_with_error, label='relative error',linewidth=2, alpha=0.6)
plt.gca().set_xscale('log')

p = norm(0, scale=0.05).rvs(size=tot)

ans_without_error = np.zeros(len(space))
ans_with_error = np.zeros(len(space))
steps = np.zeros(len(space))

for i,stop in enumerate(space):
    steps[i] = stop
    ans_without_error[i] = np.sum(w[:stop] * (1 + p[:stop]))
    ans_with_error[i] = np.sum(w[:stop])

plt.title('Likelihood Accuracy')
plt.xlabel('Number of Halos')
plt.ylabel('Likelihood')
plt.plot(steps, ans_with_error, label='likelihood',linewidth=2, alpha=0.6)
plt.plot(steps, ans_without_error, label='likelihood w error', linewidth=2,alpha=0.6)
plt.gca().set_xscale('log')
plt.gca().set_yscale('log')
plt.legend(loc=2)

plt.title('Relative Likelihood Accuracy')
plt.xlabel('Number of Halos')
plt.ylabel('Relative Error')
plt.plot(steps, np.abs(ans_without_error- ans_with_error)/ ans_with_error, label='relative error',linewidth=2, alpha=0.6)
plt.gca().set_xscale('log')


I am working on the file "MassLuminosityProject/SummerResearch/QuantifyingScalingAccuracy1_20170623.ipynb".
The first task is:
Unfortunately while the relative error in importance sampling seems iid, the weights can very dramatically and we will need to incorporate this into our model somehow. I suspect that much of this weight disparity can be explained from the mass prior. Key Assumption 2: I am going to use the variation in the mass prior probabilities for the true masses to approximately model the variation in weights of the single halo likelihoods.
Can you write Python code for it?

space = np.logspace(log10(prior.min_mass), log10(prior.max_mass))
for rs in prior.grid.redshifts:
    pdf = prior.fetch(rs).pdf(space)
    if rs == 0:
        plt.plot(space, pdf, label='z = 0')
    elif rs == 3.5:
        plt.plot(space, pdf, label='z = 3.5')
    else:
        plt.plot(space, pdf)
plt.legend(loc=3)
plt.gca().set_xscale('log')
plt.gca().set_yscale('log')
plt.xlim([prior.min_mass, prior.max_mass])
plt.title('Mass Function Prior for each Redshift Bin')
plt.xlabel('Mass ($\log_{10}(M))$')
plt.ylabel('Density')


I am working on the file "MassLuminosityProject/SummerResearch/QuantifyingScalingAccuracy1_20170623.ipynb".
The first task is:
We conclude that numerical integration is extremely accurate and that $10^5$ samples gives us 7-8 digits of precision (where we assume that computational error is monotonically decreasing with number of samples).
Importance Sampling
Can you write Python code for it?

space = np.logspace(1,6,10)
samples = 40
subplot = 1

for idx in inds:
    true_mass = data.mass.ix[idx]
    true_z = data.z.ix[idx]
    true_lum = data.lum.ix[idx]
    true_lum_obs = data.lum_obs.ix[idx]
    true_lum_obs_collection = data.lum_obs
    ans = numerical_integration(a1,a2,a3,a4,S, nsamples=10**5)
    rel_error = np.zeros((len(space), samples))
    for i,s in enumerate(space):
        for samp in xrange(samples):
            rel_error[i,samp] = (abs(importance_sampling_integration(a1,a2,a3,a4,S, nsamples=int(s)) - ans) / ans)
    shi = map(lambda x: np.percentile(x, 95), rel_error)
    hi = map(lambda x: np.percentile(x, 68), rel_error)
    mean = map(lambda x: np.percentile(x, 50), rel_error)
    lo = map(lambda x: np.percentile(x, 32), rel_error)
    slo = map(lambda x: np.percentile(x, 5), rel_error)

    plt.subplot(4,1,subplot)
    plt.plot(space, mean, linewidth=2, label='mean')
    plt.fill_between(space, lo, hi, alpha=0.4, label='1$\sigma$ envelope')
    plt.fill_between(space, slo, shi, alpha=0.2, label='2$\sigma$ envelope')
    plt.gca().set_yscale('log')
    plt.gca().set_xscale('log')
    plt.title('Importance Sampling Precision')
    plt.ylabel('Relative Error')
    plt.xlabel('Samples')
    plt.legend()
    subplot += 1

plt.gcf().set_size_inches(6,12)
plt.tight_layout()


I am working on the file "MassLuminosityProject/SummerResearch/ValidatingBigmaliAtScale_20170626.ipynb".
The first task is:
At least drawing masses from the prior has higher mean log-likelihood than from using the masses from the Millennium Simulation.
Can you write Python code for it?

print orig[:,4].mean()
print prior[:,4].mean()


I am working on the file "MassLuminosityProject/SummerResearch/ValidatingBigmaliAtScale_20170626.ipynb".
The first task is:
Failure Mode 1: The posterior is not accurate, precise, or meaningful.
Loglikelihood of Hyperseed
The point with highest likelihood does not appear to be close to the hyper-seeds (fixed hyper-parameters that generated the mock dataset). Next we compute the likelihood of the hyper-seed. We see that its log-likelihood does not fall to the right of the distribution, as we would hope.
Can you write Python code for it?

plt.title('Distribution of Log-Likelihood Weights')
plt.xlabel('Log-Likelihood')
plt.ylabel('Density')
plt.hist(prior[:,4],bins=500, normed=True, label='samples', alpha=0.5)
plt.gca().axvline(-987878.65344367269, label='hyperseed', color='r') # the likelihood of true hyper-point
plt.legend(loc=2);


I am working on the file "MassLuminosityProject/SummerResearch/ValidatingBigmaliAtScale_20170626.ipynb".
The first task is:
Failure Mode 2: The weight discrepencies are enormous and hence we end up with one sample dominating the inference.
Analyzing Weights In Each Hyperparameter Variable
Next we break up the inference samples into three equally sized buckets (low, medium, high) based on their weights. Then we plot histograms of these buckets for each hyper-parameter. Ideally we would see the good bucket adjacent to the hyper-seed and the bad bucket furthest away. This is not what we observe. We see limited difference between buckets.
It also is worth recalling the Mass-Luminosity relationship is:
\begin{align}
P(\overline{L}|\overline{M},\alpha,S,\overline{z}) &= \frac{1}{\overline{L}S\sqrt{2\pi}}\exp\left(-\frac{(\ln \overline{L} - \ln \overline{\mu_L})^2}{2S^{2}}\right)\
\mu_L &= \exp(\alpha_1) \cdot \left(\frac{M}{\alpha_3}\right)^{\alpha_2} \cdot (1+z)^{\alpha_4}\
\end{align}
Can you write Python code for it?

low = prior[np.where(prior[:,4] <= np.percentile(prior[:,4], 33))]
medium = prior[np.where((np.percentile(prior[:,4], 33) < prior[:,4]) & (prior[:,4] <= np.percentile(prior[:,4], 66)))]
high = prior[np.where(np.percentile(prior[:,4], 66) < prior[:,4])]

plt.title(r'$\alpha_1$ Histograms')
plt.hist(low[:,0], color='blue', alpha=0.5, normed=True, label='low')
plt.hist(medium[:,0], color='grey', alpha=0.5, normed=True, label='medium')
plt.hist(high[:,0], color='red', alpha=0.5, normed=True, label='high')
plt.gca().axvline(10.747809151611289, color='k', linewidth=2, label='hyper-seed')
plt.legend(loc=2)
plt.xlabel(r'$\alpha_1$')
plt.ylabel('Density');

plt.title(r'$\alpha_2$ Histograms')
plt.hist(low[:,1], color='blue', alpha=0.5, normed=True, label='low')
plt.hist(medium[:,1], color='grey', alpha=0.5, normed=True, label='medium')
plt.hist(high[:,1], color='red', alpha=0.5, normed=True, label='high')
plt.gca().axvline(0.36260141487530501, color='k', linewidth=2, label='hyper-seed')
plt.legend(loc=2)
plt.xlabel(r'$\alpha_2$')
plt.ylabel('Density');

plt.title(r'$\alpha_4$ Histograms')
plt.hist(low[:,2], color='blue', alpha=0.5, normed=True, label='low')
plt.hist(medium[:,2], color='grey', alpha=0.5, normed=True, label='medium')
plt.hist(high[:,2], color='red', alpha=0.5, normed=True, label='high')
plt.gca().axvline(1.1587242790463443, color='k', linewidth=2, label='hyper-seed')
plt.legend(loc=2)
plt.xlabel(r'$\alpha_4$')
plt.ylabel('Density');

plt.title('S Histograms')
plt.hist(low[:,3], color='blue', alpha=0.5, normed=True, label='low')
plt.hist(medium[:,3], color='grey', alpha=0.5, normed=True, label='medium')
plt.hist(high[:,3], color='red', alpha=0.5, normed=True, label='high')
plt.gca().axvline(0.1570168038792813, color='k', linewidth=2, label='hyper-seed')
plt.legend(loc=2)
plt.xlabel('S')
plt.ylabel('Density');


I am working on the file "MassLuminosityProject/SummerResearch/ValidatingBigmaliAtScale_20170626.ipynb".
The first task is:
Masses Drawn From Prior
Again, note that the weights in these corner plots are relative log-likelihood weights, as opposed to likelihood weights.
Can you write Python code for it?

corner.corner(prior[:,:4],
                       labels=[r'$\alpha_1$',r'$\alpha_2$',r'$\alpha_4$',r'$S$'],
                       weights=(prior[:,4] - prior[:,4].min()) / abs(prior[:,4].min() - prior[:,4].max()),
                       quantiles=[0.16, 0.5, 0.84],
                       show_titles=True,
                       truths=truths,
                       smooth1d=True
             );


I am working on the file "MassLuminosityProject/SummerResearch/ValidatingBigmaliAtScale_20170626.ipynb".
The first task is:
Original Masses from Millennium Simulation
Note that the weights in these corner plots are relative log-likelihood weights, as opposed to likelihood weights.
Can you write Python code for it?

corner.corner(orig[:,:4],
                       labels=[r'$\alpha_1$',r'$\alpha_2$',r'$\alpha_4$',r'$S$'],
                       weights=(orig[:,4] - orig[:,4].min()) / abs(orig[:,4].min() - orig[:,4].max()),
                       quantiles=[0.16, 0.5, 0.84],
                       show_titles=True,
                       truths=truths,
                       smooth1d=True
             );


I am working on the file "MassLuminosityProject/SummerResearch/ValidatingLikelihoodVarianceAndSingleLikelihoodWeightDistribution_20170627.ipynb".
The first task is:
And here we have the standard deviations
Can you write Python code for it?

print np.std(multi[:,4])
print np.std(multi1000[:,4])
print np.std(multi10000[:,4])


I am working on the file "MassLuminosityProject/SummerResearch/ValidatingLikelihoodVarianceAndSingleLikelihoodWeightDistribution_20170627.ipynb".
The first task is:
If we maintain our key assumption from the Quantifying Scaling Accuracy notebook, that the single log-likelihood integrals are independent and identically distributed gaussians. Then with 100 samples, the mean and variance are -8.46 and 0.06 respectively. Importance sampling is precise.
Can you write Python code for it?


I am working on the file "MassLuminosityProject/SummerResearch/ValidatingLikelihoodVarianceAndSingleLikelihoodWeightDistribution_20170627.ipynb".
The first task is:
Single Likelihood Weights
In this experiment we take 4 hypersamples and collect the weights for the single likelihood corresponding to each halo (the sum of these single log-likelihoods is the total log-likelihood).
Can you write Python code for it?

single = np.zeros((115919,4))
i = j = 0
with open('singleintegralweightvariancetest.txt', 'r') as f:
    for line in f:
        if 'likelihood' in line:
            j += 1
            i = 0
        else:
            single[i,j] = float(line)
            i += 1
f.close()

plt.hist(single[:,0], bins=50, alpha=0.4, label='l1')
plt.hist(single[:,1], bins=50, alpha=0.4, label='l2')
plt.hist(single[:,2], bins=50, alpha=0.4, label='l3')
plt.hist(single[:,3], bins=50, alpha=0.4, label='l4');
plt.title('Single Log-Likelihood Weight Histograms')
plt.ylabel('Count')
plt.xlabel('Log-Likelihood');


I am working on the file "MassLuminosityProject/SummerResearch/ValidatingLikelihoodVarianceAndSingleLikelihoodWeightDistribution_20170627.ipynb".
The first task is:
These look very similar. A natural next thing to check is the correlation between the weight sequences. We see extremely high correlation.
Can you write Python code for it?


I am working on the file "MassLuminosityProject/SummerResearch/ValidatingSimplerModel_20160626.ipynb".
The first task is:
We see that a few samples dominate the weight contribution.
Can you write Python code for it?


I am working on the file "MassSpectrometer/BackgroundSubstract.ipynb".
The first task is:
Argon
Can you write Python code for it?

Argon = pd.read_table('Ar.txt',delimiter=',  ',engine='python', header=None)

Amu = Argon[0] #These are the values of amu that the mass spec searches for

Argon = np.array([entry[:-1] for entry in Argon[1]],dtype='float')*1e6


I am working on the file "MassSpectrometer/BackgroundSubstract.ipynb".
The first task is:
Background substraction has removed a single peak at 18 amu and 17 amu, which is the water peak and OH+ fragment.
Argon Background
Kyrpton
Can you write Python code for it?

Krypton = pd.read_table('Kr.txt',delimiter=',  ',engine='python', header=None)

Krypton = np.array([entry[:-1] for entry in Krypton[1]],dtype='float')*1e6

Krbkd = pd.read_table('Background_Kr.txt',delimiter=',  ',engine='python', header=None)

Krbkd = np.array([entry[:-1] for entry in Krbkd[1]],dtype='float')*1e6

plt.figure(figsize=(9,4))
plt.scatter(Amu, Krypton - Krbkd);
ax = plt.gca()
plt.xlim(12,85);
plt.ylim(0,6)
plt.xlabel('Particle Mass [Amu]',size=18);
plt.ylabel('Pressure [Torr]$\cdot 10^{-6}$',size=18);
plt.xticks(size = 11);
plt.yticks(size = 11);
plt.savefig('Krypton.png')


I am working on the file "MassSpectrometer/BackgroundSubstract.ipynb".
The first task is:
Here, and for all subsequent measurements on this day, there is a slight peak at 40 amu, which is to be some residual from the Argon test.
Too heavy to be measured, possible for it to have shown near 40.
Neon
Can you write Python code for it?

Neon = pd.read_table('Ne.txt',delimiter=',  ',engine='python', header=None)

Neon = np.array([entry[:-1] for entry in Neon[1]],dtype='float')*1e6

Nebkd = pd.read_table('Background_Ne.txt',delimiter=',  ',engine='python', header=None)

Nebkd = np.array([entry[:-1] for entry in Nebkd[1]],dtype='float')*1e6

plt.figure(figsize=(9,4))
plt.scatter(Amu, Neon - Nebkd);
ax = plt.gca()
plt.xlim(12,35);
plt.ylim(0,3.2)
plt.xlabel('Particle Mass [Amu]',size=18);
plt.ylabel('Pressure [Torr]$\cdot 10^{-6}$',size=18);
plt.xticks(size = 11);
plt.yticks(size = 11);
plt.savefig('Neon.png')


I am working on the file "MassSpectrometer/BackgroundSubstract.ipynb".
The first task is:
Note that we were unable to gaurentee that the leak valve was in the exact same position across these two measurements. Qualitatively, we can only extract that in the quick exhale situation the ratio of N2/O2 was 4.68, while in the holding of breath it was 4.51. Not quite as expected.
Compressed Air Comparison
Can you write Python code for it?

Can1 = pd.read_table('CompressedAir_Tetrafluoroethane.txt',delimiter=',  ',engine='python', header=None)

Can1 = np.array([entry[:-1] for entry in Can1[1]],dtype='float')*1e6

Can2 = pd.read_table('CompressedAir_Difluoroethane.txt',delimiter=',  ',engine='python', header=None)

Can2 = np.array([entry[:-1] for entry in Can2[1]],dtype='float')*1e6

plt.figure(figsize=(9,4))
plt.scatter(Amu, Can1 - Quickbkd,color='blue',label='Tetrafluoroethane');
plt.scatter(Amu, Can2 - Quickbkd,color='red',label = 'Difluoroethane');
ax = plt.gca()
plt.xlim(10,65);
plt.ylim(0,8.5)
plt.xlabel('Particle Mass [Amu]',size=18);
plt.ylabel('Pressure [Torr]$\cdot 10^{-6}$',size=18);
plt.xticks(size = 11);
plt.yticks(size = 11);
plt.legend(loc='upper right')
plt.savefig('CompressedAir.png')


I am working on the file "MassSpectrometer/BackgroundSubstract.ipynb".
The first task is:
Raw Argon Data
Can you write Python code for it?

plt.figure(figsize=(9,4))
plt.scatter(Amu, Argon);
ax = plt.gca()
#ax.set_yscale('log')
plt.xlim(12,45);
plt.ylim(0,4)
plt.xlabel('Particle Mass [Amu]',size=18);
plt.ylabel('Pressure [Torr]$\cdot 10^{-6}$',size=18);
plt.xticks(size = 11);
plt.yticks(size = 11);
plt.savefig('RawArgon.png')

np.argmax(Argon)

np.argmax(Argon[0:250])

Argon[191]

Argon[391]/Argon[191]


I am working on the file "MassSpectrometer/BackgroundSubstract.ipynb".
The first task is:
Substract Argon Background
Can you write Python code for it?

Arbkd = pd.read_table('Background_Ar.txt',delimiter=',  ',engine='python', header=None)

Arbkd = np.array([entry[:-1] for entry in Arbkd[1]],dtype='float')*1e6

plt.figure(figsize=(9,4))
plt.scatter(Amu, Argon - Arbkd);
ax = plt.gca()
#ax.set_yscale('log')
plt.xlim(12,45);
plt.ylim(0,4)
plt.xlabel('Particle Mass [Amu]',size=18);
plt.ylabel('Pressure [Torr]$\cdot 10^{-6}$',size=18);
plt.xticks(size = 11);
plt.yticks(size = 11);
plt.savefig('TrueArgon.png')


I am working on the file "MasterScripts/Manuscript3_NetworkTopologyAnalysis.ipynb".
The first task is:
1.0 Basic parameters
Can you write Python code for it?

# Set basic parameters
basedir = '/projects2/ModalityControl2/'
datadir = basedir + 'data/'
resultsdir = datadir + 'resultsMaster/'
runLength = 4648

subjNums = ['032', '033', '037', '038', '039', '045',
            '013', '014', '016', '017', '018', '021',
            '023', '024', '025', '026', '027', '031',
            '035', '046', '042', '028', '048', '053',
            '040', '049', '057', '062', '050', '030', '047', '034']

glasserparcels = lg.loadGlasserParcels()
networkdef = lg.loadGlasserNetworks()
networkmappings = {'fpn':7, 'vis':1, 'smn':2, 'con':3, 'dmn':6, 'aud1':8, 'aud2':9, 'dan':11}
# Force aud2 key to be the same as aud1
aud2_ind = np.where(networkdef==networkmappings['aud2'])[0]
networkdef[aud2_ind] = networkmappings['aud1']
# Define new network mappings with no aud1/aud2 distinction
networkmappings = {'fpn':7, 'vis':1, 'smn':2, 'con':3, 'dmn':6, 'aud':8, 'dan':11}

# Define new network mappings with no aud1/aud2 distinction
networkmappings = {'fpn':7, 'vis':1, 'smn':2, 'con':3, 'dmn':6, 'aud':8, 'dan':11,
                   'prem':5, 'pcc':10, 'none':12, 'hipp':13, 'pmulti':14}

# Import network reordering
networkorder = np.asarray(sorted(range(len(networkdef)), key=lambda k: networkdef[k]))
order = networkorder
order.shape = (len(networkorder),1)
# Construct xticklabels and xticks
networks = networkmappings.keys()
xticks = {}
reorderednetworkaffil = networkdef[networkorder]
for net in networks:
    netNum = networkmappings[net]
    netind = np.where(reorderednetworkaffil==netNum)[0]
    tick = np.max(netind)
    xticks[tick] = net

from matplotlib.colors import Normalize

class MidpointNormalize(Normalize):
    def __init__(self, vmin=None, vmax=None, midpoint=None, clip=False):
        self.midpoint = midpoint
        Normalize.__init__(self, vmin, vmax, clip)

    def __call__(self, value, clip=None):
        # I'm ignoring masked values and all kinds of edge cases to make a
        # simple example...
        x, y = [self.vmin, self.midpoint, self.vmax], [0, 0.5, 1]
        return...


I am working on the file "MasterScripts/Manuscript3_NetworkTopologyAnalysis.ipynb".
The first task is:
2.0 Compute out-of-network intrinsic FC for each network (rest)
Can you write Python code for it?

# Compute rsfcMRI using multiple linear regression on Glasser parcels
fcmat_multreg = np.zeros((360,360,len(subjNums)))
scount = 0
for subj in subjNums:
    outdir = '/projects2/ModalityControl2/data/resultsMaster/MultRegConnRestFC_GlasserParcels/'
    outfile = subj + '_multregconn_restfc.csv'
    fcmat_multreg[:,:,scount] = np.loadtxt(outdir + outfile, delimiter=',')
    scount += 1


I am working on the file "MasterScripts/Manuscript3_NetworkTopologyAnalysis.ipynb".
The first task is:
Compute BGC
Can you write Python code for it?

network_gbc = {}
outofnet_gbc = {}

for net in netkeys.keys():
    network_gbc[net] = []
    outofnet_gbc[net] = []

    net_ind = np.where(networkdef==networkmappings[netkeys[net]])[0]
    outofnet_ind = np.where(networkdef!=networkmappings[netkeys[net]])[0]

    scount = 0
    for subj in subjNums:
        tmp_outofnet = []
        tmp_gbc = []
        for roi in net_ind:
            tmp_outofnet.append(np.mean(fcmat_multreg[roi,outofnet_ind,scount]))
#             tmp_outofnet.append(np.mean(fcmat_multreg[roi,:,scount]))

            tmp_gbc.append(np.mean(fcmat_multreg[roi,:,scount]))

        outofnet_gbc[net].append(np.mean(tmp_outofnet))
        network_gbc[net].append(np.mean(tmp_gbc))

        scount += 1

outofnet_avg = {}
outofnet_sem = {}
network_avg = {}
network_sem = {}
for net in netkeys.keys():
    outofnet_avg[net] = np.mean(outofnet_gbc[net])
    outofnet_sem[net] = np.std(outofnet_gbc[net])/np.sqrt(len(subjNums))
    network_avg[net] = np.mean(network_gbc[net])
    network_sem[net] = np.std(network_gbc[net])/np.sqrt(len(subjNums))

## Plot average Out-of-Network FC
# width = .35
nbars = len(netkeys)
fig = plt.figure()
ax = fig.add_subplot(111)
ax.bar(np.arange(nbars), outofnet_avg.values(), align='center',
                     yerr=outofnet_sem.values(), color='b', error_kw=dict(ecolor='black'))
ax.set_title('Average Out-of-Network (or BGC) Rest FC (MultRegFC)',
             y=1.04, fontsize=16)
ax.set_ylabel('Average FC',fontsize=12)
ax.set_xlabel('Networks', fontsize=12)
ax.set_xticks(np.arange(nbars))
ax.set_xticklabels(netkeys.values(),rotation=-45)
# ax.set_ylim([0.0025,.003])
plt.tight_layout()
# plt.savefig('Fig1_OutOfNetworkRestFC_MultReg.pdf')


I am working on the file "MasterScripts/Manuscript3_NetworkTopologyAnalysis.ipynb".
The first task is:
Interm analysis - visualize MultRegFC matrix using Glasser networks
Can you write Python code for it?

tmp = np.mean(fcmat_multreg,axis=2)
plt.figure()
norm = MidpointNormalize(midpoint=0)
plt.imshow(tmp[order,order.T], norm=norm, origin='lower', cmap='bwr', vmin=-.1, vmax=.1)
plt.colorbar(fraction=0.046)
plt.title('Averaged MultReg FC Matrix',
          fontsize=16, y=1.04)
plt.xlabel('Regions',fontsize=12)
plt.ylabel('Regions', fontsize=12)
plt.xticks(xticks.keys(),xticks.values(), rotation=-45)
plt.yticks(xticks.keys(),xticks.values())
plt.grid(linewidth=1)
plt.tight_layout()
# plt.savefig('Group_MultRegFC.pdf')


I am working on the file "MasterScripts/Manuscript3_NetworkTopologyAnalysis.ipynb".
The first task is:
Visualize Regular FC matrix using Glasser networks
Can you write Python code for it?

timeseries = np.zeros((360,360,len(subjNums)))
scount = 0
for subj in subjNums:
    indir = '/projects2/ModalityControl2/data/resultsMaster/glmRest_GlasserParcels/'
    filename = indir + subj + '_rest_nuisanceResids_Glasser.csv'
    tmp = np.loadtxt(filename, delimiter=',')
    timeseries[:,:,scount] = np.corrcoef(tmp)
    scount += 1


I am working on the file "MasterScripts/Manuscript3_NetworkTopologyAnalysis.ipynb".
The first task is:
Visualize data
Can you write Python code for it?

mat = np.mean(timeseries,axis=2)
np.fill_diagonal(mat,0)
mat = mat[order,order.T]
norm = MidpointNormalize(midpoint=0)
plt.imshow(mat,origin='lower',norm=norm, cmap='bwr', interpolation='none')
plt.colorbar(fraction=0.046)
plt.title('Averaged Pearson FC Matrix',
          fontsize=16, y=1.04)
plt.xlabel('Regions',fontsize=12)
plt.ylabel('Regions', fontsize=12)
plt.xticks(xticks.keys(),xticks.values(), rotation=-45)
plt.yticks(xticks.keys(),xticks.values())
plt.grid(linewidth=1)
plt.tight_layout()
# plt.savefig('Group_PearsonFC.pdf')


I am working on the file "MasterScripts/Manuscript4_CompModel_GroupAnalysis.ipynb".
The first task is:
3.0.1a visualize synaptic matrix (sample subject)
Can you write Python code for it?

nblocks = 20
## First four tasks are local tasks
localtasks = range(1,5)
localcommunity = 4 # local community to stimulate the local tasks
## Second four tasks are 'flexhub' tasks
flexhubtasks = range(5,9)
ntasks = len(flexhubtasks) + len(localtasks)
Tmax = 10000
Tmaxrest = 60000
#    g = 1.0
#    s = 1.0
autocorrfactor = 0
samplingrate = 1.0
TRLength=100
#### Set up subject networks ####
# Parameters for subject's networks
ncommunities = 5
innetwork_dsity = .35
outnetwork_dsity = .05
hubnetwork_dsity = .20

nodespernetwork = 50
totalnodes = nodespernetwork*ncommunities

##########
# Construct structural matrix
W = cm.generateStructuralNetwork(ncommunities=ncommunities, innetwork_dsity=innetwork_dsity,
                              outnetwork_dsity=outnetwork_dsity, hubnetwork_dsity=hubnetwork_dsity,
                              nodespernetwork=nodespernetwork, showplot=False)
# Construct synaptic matrix
G = cm.generateSynapticNetwork(W, showplot=False)
plt.figure()
# norm = MidpointNormalize(midpoint=0)
plt.imshow(G,origin='lower',interpolation='none')
plt.xlabel('Regions')
plt.ylabel('Regions')
plt.title('Synaptic Weight Matrix', y=1.04, fontsize=18)
plt.colorbar()
# plt.savefig('SingleSubj_SynapticWeightMatrix.pdf')


I am working on the file "MasterScripts/Manuscript4_CompModel_GroupAnalysis.ipynb".
The first task is:
3.2.1 Define some basic functions
Can you write Python code for it?

def runActFlowRSA((subj,net,fromnet,tasks,nblocks,fc)):
    """
    Runs a leave-block-out CV style SVM analysis (leaving 4 blocks out per CV)
    Trains on predicted ActFlow data
    Tests on real data (betas)
    """

    actflow_mat, labels = setUpActFlowRSAMat(subj,net,fromnet,tasks,nblocks=nblocks,fc=fc)
    real_mat, labels = setUpBetasRSAMat(subj,net,tasks,nblocks=nblocks)

#     actflow_mat = preprocessing.scale(actflow_mat,axis=0)
#     real_mat = preprocessing.scale(real_mat,axis=0)

    ncvs = nblocks
    indices = np.arange(actflow_mat.shape[0])
    matched_rhos = []
    mismatch_rhos = []
    for cv in range(ncvs):
        task_ind = {}
        prototype = {}
        # Construct prototypes of each task
        for task in tasks:
            # Get indices for this particular task
            task_ind[task] = np.where(labels==task)[0]
            # Decide which one is your 'comparison test trial' will be
            test_ind = task_ind[task][cv]
            # Find the indices for the prototypes
            train_ind = np.setxor1d(test_ind,task_ind[task])
            prototype[task] = np.mean(real_mat[train_ind,:],axis=0)

        # Now compare each pair of tasks with the prototype
        for task_a in tasks:
            for task_b in tasks:
                test_ind = task_ind[task_a][cv] # Compare task a
                rho_tmp = stats.spearmanr(prototype[task_b].T,actflow_mat[test_ind,:].T)[0] # With task b
                rho_tmp = np.arctanh(rho_tmp)
                if task_a==task_b:
                    # Match!
                    matched_rhos.append(rho_tmp)
                else:
                    mismatch_rhos.append(rho_tmp)

    # Get averages
    matched_rhos_avg = np.mean(matched_rhos)
    mismatch_rhos_avg = np.mean(mismatch_rhos)
    return matched_rhos_avg, mismatch_rhos_avg


I am working on the file "MasterScripts/Manuscript4_CompModel_GroupAnalysis.ipynb".
The first task is:
3.2.3 Run information transfer mapping analysis on subjects using MultReg FC
Can you write Python code for it?

# Empty variables for FlexHub task analysis
rho_mat_match_flexhub = np.zeros((ncommunities,ncommunities,nsubjs))
rho_mat_mismatch_flexhub = np.zeros((ncommunities,ncommunities,nsubjs))
for i in range(ncommunities):
    for j in range(ncommunities):
        if i==j: continue
        fromnet = i
        net = j
        nblocks = nblocks
        fc='multreg'

        ## First run on flexhub tasks
        inputs = []
        for subj in range(nsubjs): inputs.append((subj,net,fromnet,flexhubtasks,nblocks,fc))
        # Run multiprocessing
        pool = mp.Pool(processes=15)
        results_flexhub = pool.map_async(runActFlowRSA, inputs).get()
        pool.close()
        pool.join()

        ## Get results
        for subj in range(nsubjs):
            match, mismatch = results_flexhub[subj]
            rho_mat_match_flexhub[i,j,subj],rho_mat_mismatch_flexhub[i,j,subj] = match, mismatch


I am working on the file "MasterScripts/Manuscript4_CompModel_GroupAnalysis.ipynb".
The first task is:
3.2.3 Statistical testing on results and plot
Can you write Python code for it?

# Instantiate empty result matrices
tmat_flexhub = np.zeros((ncommunities,ncommunities))
pmat_flexhub = np.ones((ncommunities,ncommunities))
for i in range(ncommunities):
    for j in range(ncommunities):
        if i==j: continue
        t, p = stats.ttest_rel(rho_mat_match_flexhub[i,j,:],rho_mat_mismatch_flexhub[i,j,:])
        tmat_flexhub[i,j] = t
        # One-sided p-value
        if t > 0:
            p = p/2.0
        elif t < 0:
            p = 1.0 - p/2.0
        pmat_flexhub[i,j] = p

## FlexHub Tasks
# Run FDR correction on p-values (Don't get diagonal values)
qmat_flexhub = np.ones((ncommunities,ncommunities))
triu_ind = np.triu_indices(ncommunities,k=1)
tril_ind = np.tril_indices(ncommunities,k=-1)
all_ps = np.hstack((pmat_flexhub[triu_ind],pmat_flexhub[tril_ind]))
h, all_qs = mc.fdrcorrection0(all_ps)
# the first half of all qs belong to triu, second half belongs to tril
qmat_flexhub[triu_ind] = all_qs[:len(triu_ind[0])]
qmat_flexhub[tril_ind] = all_qs[len(tril_ind[0]):]
binary_mat_flexhub = qmat_flexhub < .05

rho_diff_mat_flexhub = np.mean(rho_mat_match_flexhub,axis=2) - np.mean(rho_mat_mismatch_flexhub,axis=2)

plt.figure()
threshold_acc = np.multiply(binary_mat_flexhub,tmat_flexhub)
norm = MidpointNormalize(midpoint=0)
plt.imshow(threshold_acc,norm=norm,origin='lower',interpolation='None',cmap='bwr')
plt.title('Network-to-Network Information Transfer\n(FDR-corrected)\nFlexHub Tasks -- MultReg FC',fontsize=16, y=1.02)
plt.colorbar()
plt.yticks(range(ncommunities), ['FlexHub', 'Net1', 'Net2', 'Net3', 'Net4'])
plt.xticks(range(ncommunities), ['FlexHub', 'Net1', 'Net2', 'Net3', 'Net4'])
plt.ylabel('Network ActFlow FROM',fontsize=15)
plt.xlabel('Network ActFlow TO',fontsize=15)
plt.tight_layout()
# plt.savefig('SFig_CompModel_Network2Network_RSA_MultRegFC_HubNetStim_.pdf')


I am working on the file "MasterScripts/Manuscript4_CompModel_GroupAnalysis.ipynb".
The first task is:
3.3 Statistical testing on results and plot using FWE-correction (permutation testing)
Can you write Python code for it?

# Instantiate empty result matrices
tfwe_flexhub = np.zeros((ncommunities,ncommunities))
pfwe_flexhub = np.ones((ncommunities,ncommunities))

ite_flexhub = rho_mat_match_flexhub - rho_mat_mismatch_flexhub

indices = np.ones((ncommunities,ncommunities))
np.fill_diagonal(indices,0)
flatten_ind = np.where(indices==1)

## FlexHub Tasks
t, p = pt.permutationFWE(ite_flexhub[flatten_ind[0],flatten_ind[1],:], permutations=1000, nproc=15)
p = 1.0 - p
np.fill_diagonal(pfwe_flexhub,1.0)
tfwe_flexhub[flatten_ind[0],flatten_ind[1]] = t
pfwe_flexhub[flatten_ind[0],flatten_ind[1]] = p
binary_mat_flexhub = pfwe_flexhub < 0.05
# Print statistics to place in text of paper
# Compute average t-value of hub-network transfers
sig_ind = pfwe_flexhub<0.05
nonsig_ind = pfwe_flexhub>0.05
print 'Average significant T-value:', np.mean(tfwe_flexhub[sig_ind])
print 'Maximum significant P-value:', np.max(pfwe_flexhub[sig_ind])
print 'Average significant ITE:', np.mean(ite_flexhub[sig_ind])
print '\n'
print 'Average non-significant T-value:', np.mean(tfwe_flexhub[nonsig_ind])
print 'Average non-significant P-value:', np.mean(pfwe_flexhub[nonsig_ind])
print 'Average non-significant ITE:', np.mean(ite_flexhub[nonsig_ind])

ite_mat_flexhub = np.mean(rho_mat_match_flexhub,axis=2) - np.mean(rho_mat_mismatch_flexhub,axis=2)

plt.figure()
threshold_acc = np.multiply(binary_mat_flexhub,tmat_flexhub)
norm = MidpointNormalize(midpoint=0)
plt.imshow(threshold_acc,norm=norm,origin='lower',interpolation='None',cmap='bwr')
plt.title('Network-to-Network Information Transfer\n(FWE-corrected)\nFlexHub Tasks -- MultReg FC',fontsize=16, y=1.02)
plt.colorbar()
plt.yticks(range(ncommunities), ['FlexHub', 'Net1', 'Net2', 'Net3', 'Net4'])
plt.xticks(range(ncommunities), ['FlexHub', 'Net1', 'Net2', 'Net3', 'Net4'])
plt.ylabel('Network ActFlow FROM',fontsize=15)
plt.xlabel('Network ActFlow TO',fontsize=15)
plt.tight_layout()
# plt.savefig('SFig_CompModel_Network2Network_RSA_MultRegFC_HubNetStim_.pdf')


I am working on the file "MasterScripts/Manuscript4_CompModel_GroupAnalysis.ipynb".
The first task is:
Correct for FWE instead
Can you write Python code for it?

contrast = np.zeros((ncommunities-1,outofnet_intrinsicFC.shape[1]))

hubnet = 0
i = 0
for net in range(ncommunities):
    if hubnet == net: continue
    t, p = stats.ttest_rel(outofnet_intrinsicFC[hubnet,:],outofnet_intrinsicFC[net,:])
    contrast[i,:] = outofnet_intrinsicFC[hubnet,:] - outofnet_intrinsicFC[net,:]
    i += 1

t, p_fwe = pt.permutationFWE(contrast, permutations=1000, nproc=15)
p_fwe = 1.0 - p_fwe

qvals = mc.fdrcorrection0(pvals)[1]

for net in range(ncommunities):
    if net == hubnet:
        print 'Average out-of-network GBC of network', net, ':', round(np.mean(outofnet_intrinsicFC[net,:]),5)
    else:
        print 'Average out-of-network GBC of network', net, ':', round(np.mean(outofnet_intrinsicFC[net,:]),5), '\t t =', round(t[net-1],3), '\t p =', round(p_fwe[net-1],3)


I am working on the file "MasterScripts/Manuscript5_RegionInformationEstimate_CPRORules.ipynb".
The first task is:
0.0 Basic parameters
Can you write Python code for it?

# Set basic parameters
basedir = '/projects2/ModalityControl2/'
datadir = basedir + 'data/'
resultsdir = datadir + 'resultsMaster/'
runLength = 4648

subjNums = ['032', '033', '037', '038', '039', '045',
            '013', '014', '016', '017', '018', '021',
            '023', '024', '025', '026', '027', '031',
            '035', '046', '042', '028', '048', '053',
            '040', '049', '057', '062', '050', '030', '047', '034']

# Organized as a 64k vector
glasserparcels = lg.loadGlasserParcels()

nParcels = 360

# Load in Glasser parcels in their native format
# Note that this parcel file is actually flipped (across hemispheres), but it doesn't matter since we're using the same exact file to reconstruct the data
glasser2 = nib.load('/projects/AnalysisTools/ParcelsGlasser2016/archive/Q1-Q6_RelatedParcellation210.LR.CorticalAreas_dil_Colors.32k_fs_LR.dlabel.nii')
glasser2 = glasser2.get_data()
glasser2 = np.squeeze(glasser2)


I am working on the file "MasterScripts/Manuscript5_RegionInformationEstimate_CPRORules.ipynb".
The first task is:
2.0 - Estimate information estimates for all regions for all 3 rule domains
Can you write Python code for it?

ruledims = ['logic','sensory','motor']
behav='acc'
corr_rhos = {}
err_rhos = {}
diff_rhos = {}
for ruledim in ruledims:
    corr_rhos[ruledim] = {}
    err_rhos[ruledim] = {}
    diff_rhos[ruledim] = {}

    print 'Running', ruledim

    inputs = []
    for subj in subjNums: inputs.append((subj,ruledim,behav))

#     pool = mp.Pool(processes=8)
    pool = mp.Pool(processes=16)
    results = pool.map_async(subjRSACV,inputs).get()
    pool.close()
    pool.join()

    # Reorganize results
    corr_rhos[ruledim] = np.zeros((nParcels,len(subjNums)))
    err_rhos[ruledim] = np.zeros((nParcels,len(subjNums)))
    diff_rhos[ruledim] = np.zeros((nParcels,len(subjNums)))

    scount = 0
    for result in results:
        for roi in range(nParcels):
            corr_rhos[ruledim][roi,scount] = result[0][roi+1]
            err_rhos[ruledim][roi,scount] = result[1][roi+1]
            diff_rhos[ruledim][roi,scount] = result[2][roi+1]
        scount += 1


I am working on the file "MasterScripts/Manuscript5_RegionInformationEstimate_CPRORules.ipynb".
The first task is:
Map statistics and results to surface using workbench
Can you write Python code for it?

sig_t_vertex = np.zeros((len(glasser2),len(ruledims)))
effects_vertex = np.zeros((len(glasser2),len(ruledims)))
effects_vertex_sig = np.zeros((len(glasser2),len(ruledims)))
col = 0
for cols in range(sig_t_vertex.shape[1]):
    for roi in range(nParcels):
        parcel_ind = np.where(glasser2==(roi+1))[0]
        sig_t_vertex[parcel_ind,col] = sig_t[roi,col]
        effects_vertex[parcel_ind,col] = effectsize[ruledims[col]][roi]
        effects_vertex_sig[parcel_ind,col] = sig_effect[roi,col]
    col += 1

# Write file to csv and run wb_command
outdir = '/projects2/ModalityControl2/data/resultsMaster/Manuscript5_BaselineRegionIE/'
filename = 'RegionwiseIE_FDRThresholded_Tstat.csv'
np.savetxt(outdir + filename, sig_t_vertex,fmt='%s')
wb_file = 'RegionwiseIE_FDRThresholded_Tstat.dscalar.nii'
glasserfilename = '/projects/AnalysisTools/ParcelsGlasser2016/archive/Q1-Q6_RelatedParcellation210.LR.CorticalAreas_dil_Colors.32k_fs_LR.dlabel.nii'
wb_command = 'wb_command -cifti-convert -from-text ' + outdir + filename + ' ' + glasserfilename + ' ' + outdir + wb_file + ' -reset-scalars'
os.system(wb_command)

# Compute effect size baseline (information content)
outdir = '/projects2/ModalityControl2/data/resultsMaster/Manuscript5_BaselineRegionIE/'
filename = 'RegionwiseIE_InformationEstimate.csv'
np.savetxt(outdir + filename, effects_vertex,fmt='%s')
wb_file = 'RegionwiseIE_InformationEstimate.dscalar.nii'
wb_command = 'wb_command -cifti-convert -from-text ' + outdir + filename + ' ' + glasserfilename + ' ' + outdir + wb_file + ' -reset-scalars'
os.system(wb_command)

# Compute Thresholded effect size baseline (information content)
outdir = '/projects2/ModalityControl2/data/resultsMaster/Manuscript5_BaselineRegionIE/'
filename = 'RegionwiseIE_FDRThresholded_InformationEstimate.csv'
np.savetxt(outdir + filename, effects_vertex_sig,fmt='%s')
wb_file = 'RegionwiseIE_FDRThresholded_InformationEstimate.dscalar.nii'
wb_command = 'wb_command -cifti-convert -from-text ' + outdir + filename + ' ' + glasserfilename + ' ' + outdir...


I am working on the file "MasterScripts/Manuscript5_RegionInformationEstimate_CPRORules.ipynb".
The first task is:
Run statistics (t-tests and multiple comparisons correction with FDR)
Can you write Python code for it?

df_stats = {}
# Output to CSV matrix
sig_t = np.zeros((nParcels,len(ruledims)))
sig_effect = np.zeros((nParcels,len(ruledims)))
effectsize = {}
rulecount = 0
for ruledim in ruledims:
    df_stats[ruledim] = {}
    df_stats[ruledim]['t'] = np.zeros((nParcels,))
    df_stats[ruledim]['p'] = np.zeros((nParcels,))
    effectsize[ruledim] = np.zeros((nParcels,))
    for roi in range(nParcels):
        t, p = stats.ttest_1samp(diff_rhos[ruledim][roi,:], 0)
#         t, p = stats.ttest_rel(corr_rhos[ruledim][roi,:], err_rhos[ruledim][roi,:])

        effectsize[ruledim][roi] = np.mean(diff_rhos[ruledim][roi,:])

        ps = np.zeros(())
        if t > 0:
            p = p/2.0
        else:
            p = 1.0 - p/2.0
        df_stats[ruledim]['t'][roi] = t
        df_stats[ruledim]['p'][roi] = p

    arr = df_stats[ruledim]['p']
    df_stats[ruledim]['q'] = mc.fdrcorrection0(arr)[1]

    qbin = df_stats[ruledim]['q'] < 0.05
    sig_t[:,rulecount] = np.multiply(df_stats[ruledim]['t'],qbin)
    sig_effect[:,rulecount] = np.multiply(effectsize[ruledim],qbin)

    rulecount += 1


I am working on the file "MasterScripts/Manuscript5_RegionInformationEstimate_CPRORules.ipynb".
The first task is:
Save CSVs for baseline leave-4-out CV on information transfer estimate
Can you write Python code for it?

outdir = '/projects2/ModalityControl2/data/resultsMaster/Manuscript5_BaselineRegionIE/'

for ruledim in ruledims:
    filename = 'Regionwise_InformationEstimate_LeaveOneOut_' + ruledim + '.csv'
    np.savetxt(outdir + filename, diff_rhos[ruledim], delimiter=',')


I am working on the file "MasterScripts/Manuscript5_RegionInformationEstimate_CPRORules.ipynb".
The first task is:
Write out significant ROIs
Can you write Python code for it?

fwe_Ps2 = (1.0000 - fwe_Ps) # One-tailed test on upper tail
sig_mat = fwe_Ps2 < 0.0500 # One-sided t-test (Only interested in values greater than 95% interval)
outdir = '/projects2/ModalityControl2/data/resultsMaster/Manuscript5_BaselineRegionIE/'
filename = 'FWE_corrected_pvals_allruledims.csv'
np.savetxt(outdir + filename, fwe_Ps, delimiter=',')

sig_t = np.zeros((nParcels,len(ruledims)))
sig_effect = np.zeros((nParcels,len(ruledims)))
rulecount = 0
for ruledim in ruledims:
    sig_t[:,rulecount] = np.multiply(fwe_Ts[:,rulecount],sig_mat[:,rulecount])
    sig_effect[:,rulecount] = np.multiply(effectsize[ruledim],sig_mat[:,rulecount])

    # Read out statistics for manuscript
    # Identify significant regions
    sig_ind = sig_mat[:,rulecount] == True
    nonsig_ind = sig_mat[:,rulecount] == False

    print 'Average significant IE for', ruledim, ':', np.mean(effectsize[ruledim][sig_ind])
    print 'Average significant T-stats for', ruledim, ':', np.mean(fwe_Ts[:,rulecount][sig_ind])
    print 'Maximum significant p-value for', ruledim, ':', np.max(fwe_Ps2[:,rulecount][sig_ind])
    print '----'
    print 'Average nonsignificant IE for', ruledim, ':', np.mean(effectsize[ruledim][nonsig_ind])
    print 'Average nonsignificant T-stats for', ruledim, ':', np.mean(fwe_Ts[:,rulecount][nonsig_ind])
    print 'Minimum nonsignificant p-value for', ruledim, ':', np.min(fwe_Ps2[:,rulecount][nonsig_ind])
    print '\n'
    print '*****************'
    rulecount += 1


I am working on the file "MasterScripts/Manuscript7_PercentFROM_Region_ITE.ipynb".
The first task is:
0.0 Basic parameters
Can you write Python code for it?

# Set basic parameters
basedir = '/projects2/ModalityControl2/'
datadir = basedir + 'data/'
resultsdir = datadir + 'resultsMaster/'
runLength = 4648

subjNums = ['032', '033', '037', '038', '039', '045',
            '013', '014', '016', '017', '018', '021',
            '023', '024', '025', '026', '027', '031',
            '035', '046', '042', '028', '048', '053',
            '040', '049', '057', '062', '050', '030', '047', '034']

glasserparcels = lg.loadGlasserParcels()
networkdef = lg.loadGlasserNetworks()

# Define the main networks (in main manuscript)
networkmappings = {'fpn':7, 'vis':1, 'smn':2, 'con':3, 'dmn':6, 'aud':8, 'aud2':9, 'dan':11}
# Force aud2 key to be the same as aud1
# aud2_ind = np.where(networkdef==networkmappings['aud2'])[0]
# networkdef[aud2_ind] = networkmappings['aud1']
# Merge aud1 and aud2
# networkmappings = {'fpn':7, 'vis':1, 'smn':2, 'con':3, 'dmn':6, 'aud':8, 'dan':11}

nParcels = 360

# Import network reordering
networkorder = np.asarray(sorted(range(len(networkdef)), key=lambda k: networkdef[k]))
order = networkorder
order.shape = (len(networkorder),1)

# Construct xticklabels and xticks for plotting figures
networks = networkmappings.keys()
xticks = {}
reorderednetworkaffil = networkdef[order]
for net in networks:
    netNum = networkmappings[net]
    netind = np.where(reorderednetworkaffil==netNum)[0]
    tick = np.max(netind)
    xticks[tick] = net

# Load in Glasser parcels in their native format (vertex formula)
glasserfilename = '/projects/AnalysisTools/ParcelsGlasser2016/archive/Q1-Q6_RelatedParcellation210.LR.CorticalAreas_dil_Colors.32k_fs_LR.dlabel.nii'
glasser2 = nib.load('/projects/AnalysisTools/ParcelsGlasser2016/archive/Q1-Q6_RelatedParcellation210.LR.CorticalAreas_dil_Colors.32k_fs_LR.dlabel.nii')
glasser2 = np.squeeze(glasser2.get_data())


I am working on the file "MasterScripts/Manuscript7_PercentFROM_Region_ITE.ipynb".
The first task is:
2.2 Compute the regions with the most information transfers TO and FROM
Can you write Python code for it?

networks = networkmappings.keys()

regions_actflowTO = {}
regions_actflowFROM = {}

for ruledim in ruledims:

    thresh = df_stats[ruledim]['q'] < 0.05
    regions_actflowFROM[ruledim] = np.nanmean(thresh,axis=1)*100.0
    regions_actflowTO[ruledim] = np.nanmean(thresh,axis=0)*100.0

#     # Convert to dataframe
#     plt.figure()
#     plt.bar(np.arange(nParcels),regions_actflow[ruledim],align='center')
#     plt.title('Percent of Significant ActFlow FROM each region', fontsize=16)
#     plt.ylabel('Percent of Significant ActFlow\nTo Other Regions', fontsize=12)
#     plt.xlabel('Regions', fontsize=12)
#     plt.tight_layout()

# Save these arrays to a file
savearrayTO = np.zeros((len(glasser2),len(ruledims)+1))
savearrayFROM = np.zeros((len(glasser2),len(ruledims)+1))

rulecount = 0
for ruledim in ruledims:
    for roi in range(1,nParcels+1):
        parcel_ind = np.where(glasser2==roi)[0]
        # Compute map of all rule dimension for rule general actflow
        if rulecount < 3:
            savearrayTO[parcel_ind,rulecount] = regions_actflowTO[ruledim][roi-1].astype('double')
            savearrayFROM[parcel_ind,rulecount] = regions_actflowFROM[ruledim][roi-1].astype('double')

    rulecount += 1

to_avg = savearrayTO[:,0:3] > 0
# Create conjunction map
to_avg = np.mean(to_avg,axis=1)
to_avg = (to_avg == 1)
savearrayTO[:,3] = to_avg

from_avg = savearrayFROM[:,0:3] > 0
from_avg = np.mean(from_avg,axis=1)
from_avg = (from_avg == 1)
savearrayFROM[:,3] = from_avg

outdir = '/projects2/ModalityControl2/data/resultsMaster/Manuscript6andS2and7_RegionToRegionITE/'
filename = 'PercentOfRegionsSignificantActFlowFROM_FDR.csv'
np.savetxt(outdir + filename,savearrayFROM,fmt='%s')
wb_file = 'PercentOfRegionsSignificantActFlowFROM_FDR.dscalar.nii'
wb_command = 'wb_command -cifti-convert -from-text ' + outdir + filename + ' ' + glasserfilename + ' ' + outdir + wb_file + ' -reset-scalars'
os.system(wb_command)

outdir =...


I am working on the file "MasterScripts/Manuscript7_PercentFROM_Region_ITE.ipynb".
The first task is:
3.0 Compute FWE-corrected results (as opposed to FDR)
Can you write Python code for it?

## Load in NM3 Data
ruledims = ['logic','sensory','motor']
datadir = '/projects2/ModalityControl2/data/resultsMaster/Manuscript6andS2and7_RegionToRegionITE/'
# Load in RSA matrices
iteMats = {}
df_stats = {}
for ruledim in ruledims:
    iteMats[ruledim] = np.zeros((nParcels,nParcels,len(subjNums)))
    df_stats[ruledim] = {}
    scount = 0
    for subj in subjNums:
        filename = datadir +subj+'_' + ruledim + '_RegionToRegionActFlowGlasserParcels.csv'
        iteMats[ruledim][:,:,scount] = np.loadtxt(filename, delimiter=',')
        scount += 1

pt = reload(pt)

fwe_Ts = np.zeros((nParcels,nParcels,len(ruledims)))
fwe_Ps = np.zeros((nParcels,nParcels,len(ruledims)))

# Obtain indices for multiple comparisons
indices = np.ones((nParcels,nParcels))
np.fill_diagonal(indices,0)
notnan_ind = np.isnan(iteMats['logic'][:,:,0])==False
indices = np.multiply(indices,notnan_ind)
flatten_ind = np.where(indices==1)
rulecount = 0

for ruledim in ruledims:
#     tmpcor = np.arctanh(corrMats[ruledim][flatten_ind[0],flatten_ind[1],:])
#     tmperr = np.arctanh(errMats[ruledim][flatten_ind[0],flatten_ind[1],:])
    t, p = pt.permutationFWE(iteMats[ruledim][flatten_ind[0],flatten_ind[1],:], permutations=1000, nproc=15)
    fwe_Ts[flatten_ind[0],flatten_ind[1],rulecount] = t
    fwe_Ps[flatten_ind[0],flatten_ind[1],rulecount] = 1.0 - p

    rulecount += 1


I am working on the file "MasterScripts/Manuscript7_PercentFROM_Region_ITE.ipynb".
The first task is:
3.1 Visualize information transfer mapping matrices (FWE-Threshold and Unthresholded)
Can you write Python code for it?

pthresh = .05

# Visualize FWER-corrected T-statistic map
order = networkorder
order.shape = (len(networkorder),1)
rulecount = 0
for ruledim in ruledims:

    # Thresholded T-stat map
    plt.figure()
    # First visualize unthresholded
    mat = fwe_Ts[:,:,rulecount]
    thresh = fwe_Ps[:,:,rulecount] < pthresh
    mat = np.multiply(mat,thresh)
    mat = mat[order,order.T]
    ind = np.isnan(mat)
    mat[ind]=0
    pos = mat > 0
    mat = np.multiply(pos,mat)
    norm = MidpointNormalize(midpoint=0)
    plt.imshow(mat,origin='lower',norm=norm,vmin = 0,interpolation='none',cmap='seismic')
    plt.colorbar(fraction=0.046)
    plt.title('FWE-corrected T-statistic Map\nInformation Transfer Estimates\n ' + ruledim,
              fontsize=16, y=1.04)
    plt.xlabel('Target Regions',fontsize=12)
    plt.ylabel('Source Regions', fontsize=12)
    plt.xticks(xticks.keys(),xticks.values(), rotation=-45)
    plt.yticks(xticks.keys(),xticks.values())
    plt.grid(linewidth=1)
    plt.tight_layout()

#     plt.savefig('Fig6_RegionITE_TstatMap' + ruledim + '_FWER.pdf')

    rulecount += 1


I am working on the file "MasterScripts/Manuscript7_PercentFROM_Region_ITE.ipynb".
The first task is:
3.2 Compute the regions with the most information transfers TO and FROM
Can you write Python code for it?

networks = networkmappings.keys()

regions_actflowTO = {}
regions_actflowFROM = {}

rulecount = 0
for ruledim in ruledims:

    thresh = fwe_Ps[:,:,rulecount] > pthresh
    regions_actflowFROM[ruledim] = np.nanmean(thresh,axis=1)*100.0
    regions_actflowTO[ruledim] = np.nanmean(thresh,axis=0)*100.0
    rulecount += 1

# Save these arrays to a file
savearrayTO = np.zeros((len(glasser2),len(ruledims)+1))
savearrayFROM = np.zeros((len(glasser2),len(ruledims)+1))

rulecount = 0
for ruledim in ruledims:
    for roi in range(1,nParcels+1):
        parcel_ind = np.where(glasser2==roi)[0]
        # Compute map of all rule dimension for rule general actflow
        if rulecount < 3:
            savearrayTO[parcel_ind,rulecount] = regions_actflowTO[ruledim][roi-1].astype('double')
            savearrayFROM[parcel_ind,rulecount] = regions_actflowFROM[ruledim][roi-1].astype('double')

    rulecount += 1

to_avg = savearrayTO[:,0:3] > 0
# Create conjunction map
to_avg = np.mean(to_avg,axis=1)
to_avg = (to_avg == 1)
savearrayTO[:,3] = to_avg

from_avg = savearrayFROM[:,0:3] > 0
from_avg = np.mean(from_avg,axis=1)
from_avg = (from_avg == 1)
savearrayFROM[:,3] = from_avg

outdir = '/projects2/ModalityControl2/data/resultsMaster/Manuscript6andS2and7_RegionToRegionITE/'
filename = 'PercentOfRegionsSignificantActFlowFROM_FWER.csv'
np.savetxt(outdir + filename,savearrayFROM,fmt='%s')
wb_file = 'PercentOfRegionsSignificantActFlowFROM_FWER.dscalar.nii'
wb_command = 'wb_command -cifti-convert -from-text ' + outdir + filename + ' ' + glasserfilename + ' ' + outdir + wb_file + ' -reset-scalars'
os.system(wb_command)

outdir = '/projects2/ModalityControl2/data/resultsMaster/Manuscript6andS2and7_RegionToRegionITE/'
filename = 'PercentOfRegionsSignificantActFlowTO_FWER.csv'
np.savetxt(outdir + filename,savearrayTO,fmt='%s')
wb_file = 'PercentOfRegionsSignificantActFlowTO_FWER.dscalar.nii'
wb_command = 'wb_command -cifti-convert -from-text ' + outdir + filename + ' ' + glasserfilename + ' ' + outdir + wb_file + '...


I am working on the file "MasterScripts/ManuscriptS1a_NetworkInformationEstimate_Supplementary.ipynb".
The first task is:
Run statistics with FDR-correction
Can you write Python code for it?

# Compute group stats
chance = 0.0
results_dict_fdr = {}
for ruledim in ruledims:
    results_dict_fdr[ruledim] = {}
    for net in netkeys.keys(): results_dict_fdr[ruledim][netkeys[net]] = {}
    pvals = []
    for net in netkeys.keys():
        results_dict_fdr[ruledim][netkeys[net]]['Accuracy'] = str(round(np.mean(avg_acc[ruledim][net]),3))
        t, p = stats.ttest_1samp(diff_rho[ruledim][net],chance)
        results_dict_fdr[ruledim][netkeys[net]]['T-stats'] = t
        results_dict_fdr[ruledim][netkeys[net]]['P-values'] = p
        pvals.append(p)

    qvals = mc.fdrcorrection0(pvals)[1]
    qcount = 0
    for net in netkeys.keys():
        results_dict_fdr[ruledim][netkeys[net]]['Q-values'] = qvals[qcount]
        qcount += 1


I am working on the file "MasterScripts/ManuscriptS1a_NetworkInformationEstimate_Supplementary.ipynb".
The first task is:
Run statistics with FWER-correction (Permutation testing)
Can you write Python code for it?

# Compute group stats
chance = 0.0
results_dict_fwe = {}
for ruledim in ruledims:
    results_dict_fwe[ruledim] = {}
    for net in netkeys.keys(): results_dict_fwe[ruledim][netkeys[net]] = {}
    pvals = []
    for net in netkeys.keys():
        results_dict_fwe[ruledim][netkeys[net]]['Accuracy'] = str(round(np.mean(avg_acc[ruledim][net]),3))
        t, p = pt.permutationFWE(diff_rho[ruledim],nullmean=0,permutations=10000,nproc=15)
#         t, p = stats.ttest_1samp(diff_rho[ruledim][net],chance)
        results_dict_fwe[ruledim][netkeys[net]]['T-stats'] = t[net]
        results_dict_fwe[ruledim][netkeys[net]]['P-FWE'] = 1.0 - p[net]


I am working on the file "MasterScripts/ManuscriptS1a_NetworkInformationEstimate_Supplementary.ipynb".
The first task is:
Set up functions for information estimation (RSA instead of SVM decoding)
Can you write Python code for it?

def rsaCV(svm_mat,labels,ruledim):
    """Runs a leave-4-out CV for a 4 way  classification"""

    cvfolds = []
    # 32 folds, if we do a leave 4 out for 128 total miniblocks
    # Want to leave a single block from each rule from each CV
    for rule in np.unique(labels):
        cvfolds.append(np.where(labels==rule)[0])
    cvfolds = np.asarray(cvfolds)

    # Number of CVs is columns
    ncvs = cvfolds.shape[1]
    nrules = cvfolds.shape[0]
    # Randomly sample cross-validation folds
    for i in range(nrules): np.random.shuffle(cvfolds[i,:])

    corr_rho_cvs = []
    err_rho_cvs = []
    for cv in range(ncvs):
        # Select a test set from the CV Fold matrix
        test_ind = cvfolds[:,cv].copy()
        # Delete the CV included from the train set
        train_ind = np.delete(cvfolds,cv,axis=1)

        # Identify the train and test sets
        svm_train = svm_mat[np.reshape(train_ind,-1),:]
        svm_test = svm_mat[test_ind,:]

#         ## Feature-wise normalization (computed by hand)
#         # Compute mean of train set
#         train_mean = np.mean(svm_train,axis=0)
#         train_mean.shape = (1,len(train_mean))
#         # Compute std of train set
#         train_std = np.std(svm_train,axis=0)
#         train_std.shape = (1,len(train_std))
#         # Normalize train set
#         svm_train = np.divide((svm_train - train_mean),train_std)
#         # Normalize test set with trainset mean and std to avoid circularity
#         svm_test = (svm_test - train_mean)/train_std

        prototype = {}
        # Construct RSA prototypes
        for rule in range(nrules):
            prototype_ind = np.reshape(train_ind[rule,:],-1)
            prototype[rule] = np.mean(svm_mat[prototype_ind],axis=0)

        corr_rho = []
        err_rho = []
        for rule1 in range(nrules):
            for rule2 in range(nrules):
                r = stats.spearmanr(prototype[rule1],svm_test[rule2,:])[0]
                r = np.arctanh(r)
                if...


I am working on the file "MasterScripts/ManuscriptS1a_NetworkInformationEstimate_Supplementary.ipynb".
The first task is:
Show results as dataframe
Can you write Python code for it?

results_dframe_fwe = {}
for ruledim in ruledims:
    print 'Dataframe for', ruledim, 'classification'
    results_dframe_fwe[ruledim] = pandas.DataFrame(data=results_dict_fwe[ruledim])
    display(results_dframe_fwe[ruledim])


I am working on the file "MasterScripts/ManuscriptS1b_Network2NetworkInformationTransfer.ipynb".
The first task is:
1.0 Compute network-to-network activity flow mapping
1.1 Set up basic ActFlow functions
Can you write Python code for it?

def computeRestFC(subj):
    """
    Computes the Glasser parcellated restFC of a given subject
    """
    filename = resultsdir + 'glmRest_GlasserParcels/' + subj + '_rest_nuisanceResids_Glasser.csv'
    timeseries = np.loadtxt(filename, delimiter=',')
    fcmat = mreg.multregressionconnectivity(timeseries)
    return fcmat

def loadBetas(subj, net='all'):
    """Loads in task betas"""
    datafile = resultsdir + 'glmMiniblockBetaSeries/' + subj + '_miniblock_taskBetas_Glasser.csv'
    betas = np.loadtxt(datafile, delimiter=',')
    betas = betas[:,17:]
    if net == 'all':
        return betas.T
    else:
        net_ind = np.where(networkdef==net)[0]
        return betas[net_ind,:].T

def computeNetworkToNetworkActFlow(subj,fcmat):
    """
    Given a subject number and a resting-state FC Matrix, compute all possible network-to-network ActFlow predictions
    """
    outdir = resultsdir + 'ManuscriptS1_NetworkIEandITE_Analysis/ActFlowNetworkToNetwork/'
    betas  = loadBetas(subj, net='all')

    for net1 in networkmappings:
        for net2 in networkmappings:
            if net1==net2: continue
#             print 'ActFlow:', net1, 'to', net2
            # Find parcels for net1 and reshape
            net1_ind = np.where(networkdef==networkmappings[net1])[0]
            net1_ind.shape = (len(net1_ind),1)
            # Find parcels for net2 and reshape
            net2_ind = np.where(networkdef==networkmappings[net2])[0]
            net2_ind.shape = (len(net2_ind),1)

            # Compute betas for net1 and net2
            net1beta = np.squeeze(betas[:,net1_ind.T]) # Reformat to miniblocks x parcels
            net2beta = np.squeeze(betas[:,net2_ind.T]) # Reformat to miniblocks x parcels

            # Compute FC for net1 to net2
            w12 = fcmat[net1_ind,net2_ind.T].copy()
            w21 = fcmat[net2_ind,net1_ind.T].copy()

            # Randomize topology
#             np.random.shuffle(w12)
#             np.random.shuffle(w21)

        ...


I am working on the file "MasterScripts/ManuscriptS1b_Network2NetworkInformationTransfer.ipynb".
The first task is:
1.2 Estimate FC using multiple linear regression
1.2.1 Run MultRegConnFC (rest) and save to disk since it takes a while
Can you write Python code for it?

for subj in subjNums:
    print 'Computing MultRegConn FC for subject', subj
    fcmat = computeRestFC(subj)
    outdir = resultsdir + '/MultRegConnRestFC_GlasserParcels/'
    outfile = subj + '_multregconn_restfc.csv'
    np.savetxt(outdir + outfile, fcmat, delimiter=',')


I am working on the file "MasterScripts/ManuscriptS1b_Network2NetworkInformationTransfer.ipynb".
The first task is:
1.3 Run network-to-network activity flow mapping
Can you write Python code for it?

for subj in subjNums:
#     print 'Running ActFlow on subject', subj
    ActFlowSubj(subj)


I am working on the file "MasterScripts/ManuscriptS1b_Network2NetworkInformationTransfer.ipynb".
The first task is:
2.0 Predicted-to-actual similarity analysis on activity flow predictions
2.0.1 Set up basic functions for ActFlowStatistics
Can you write Python code for it?

def getActFlowActivity(subj, fromnet, tonet):
    datadir = resultsdir + 'ManuscriptS1_NetworkIEandITE_Analysis/ActFlowNetworkToNetwork/'
    datafile = datadir+subj+'_'+fromnet+'TO'+tonet+'_actflow_GlasserParcels.csv'
    activity = np.loadtxt(datafile, delimiter=',')
    return activity

def setUpActFlowMatrix(subj,ruledim,fromnet,tonet):
    """
    Sets up basic SVM matrix for a classification of a particular rule dimension and network for ActFlow
    """
    activity = getActFlowActivity(subj,fromnet,tonet)
    rules, rulesmb = func.importRuleTimingsV3(subj,ruledim)

    svm_mat = np.zeros((activity.shape))
    samplecount = 0
    labels = []
    for rule in rulesmb:
        rule_ind = rulesmb[rule].keys()
        sampleend = samplecount + len(rule_ind)
        svm_mat[samplecount:sampleend,:] = activity[rule_ind,:]
        labels.extend(np.ones(len(rule_ind),)*rule)
        samplecount += len(rule_ind)

    labels = np.asarray(labels)

    return svm_mat, labels

def setUpRSAMatrix(subj,ruledim,net):
    """
    Sets up basic SVM Matrix for a classification of a particular rule dimension and network
    """

    betas = loadBetas(subj,net=net)
    rules, rulesmb = func.importRuleTimingsV3(subj,ruledim)

    svm_mat = np.zeros((betas.shape))
    samplecount = 0
    labels = []
    for rule in rulesmb:
        rule_ind = rulesmb[rule].keys()
        sampleend = samplecount + len(rule_ind)
        svm_mat[samplecount:sampleend,:] = betas[rule_ind,:]
        labels.extend(np.ones(len(rule_ind),)*rule)
        samplecount += len(rule_ind)

    labels = np.asarray(labels)

    return svm_mat, labels


I am working on the file "MasterScripts/ManuscriptS1b_Network2NetworkInformationTransfer.ipynb".
The first task is:
2.2 Run predicted-to-actual similarity (RSA) using multiprocessing
Keep track of networks to matrix indices
Can you write Python code for it?

# Remake order to fit that of the glasser community ordering (region-to-region matrix ordering)
netkeys = {0:'vis',1:'smn',2:'con',3:'dmn',4:'fpn', 5:'aud', 6:'dan'}
print netkeys
num_networks=len(netkeys)

baseline = 0.0
ruledims = ['logic','sensory','motor']
numnets = len(networkmappings)
output = {}
avg_rho = {}
tstats = {}
pvals = {}
corr_rho = {}
err_rho = {}
diff_rho ={}
for ruledim in ruledims:
    output[ruledim] = np.zeros((num_networks,num_networks,len(subjNums)))
    avg_rho[ruledim] = np.zeros((num_networks,num_networks))
    tstats[ruledim] = np.zeros((num_networks,num_networks))
    pvals[ruledim] = np.zeros((num_networks,num_networks))
    corr_rho[ruledim] = np.zeros((num_networks,num_networks,len(subjNums)))
    err_rho[ruledim] = np.zeros((num_networks,num_networks,len(subjNums)))
    diff_rho[ruledim] = np.zeros((num_networks,num_networks,len(subjNums)))

    print 'Running', ruledim
    for net1 in range(num_networks):

        for net2 in range(num_networks):
            # Skip if net1 and net2
            if net1==net2:
                avg_rho[ruledim][net1,net2] = np.nan
                tstats[ruledim][net1,net2] = np.nan
                pvals[ruledim][net1,net2] = np.nan
                corr_rho[ruledim][net1,net2] = np.nan
                err_rho[ruledim][net1,net2] = np.nan
                diff_rho[ruledim][net1,net2] = np.nan
                continue

            inputs = []
            for subj in subjNums: inputs.append((subj,ruledim,netkeys[net1],netkeys[net2]))
            # Run in parallel
            pool = mp.Pool(processes=16)
            results = pool.map_async(subjRSACV,inputs).get()
            pool.close()
            pool.join()

            i = 0
            for result in results:

                tmp_corr, tmp_err, tmp_diff = result
                corr_rho[ruledim][net1,net2,i] = tmp_corr
                err_rho[ruledim][net1,net2,i] = tmp_err
                diff_rho[ruledim][net1,net2,i] = tmp_diff
            ...


I am working on the file "MasterScripts/ManuscriptS1b_Network2NetworkInformationTransfer.ipynb".
The first task is:
2.3 Compute group statistics
Can you write Python code for it?

# Compute group stats
baseline = 0.0
triu_indices = np.triu_indices(len(networkmappings),k=1)
tril_indices = np.tril_indices(len(networkmappings),k=-1)

qmat = {}
sigflow = {}
sig_rho = {}
tstat_sig = {}
for ruledim in ruledims:

    qmat[ruledim] = np.zeros((num_networks,num_networks))
    sigflow[ruledim] = np.zeros((num_networks,num_networks))
    sig_rho[ruledim] = np.zeros((num_networks,num_networks))
    tstat_sig[ruledim] = np.zeros((num_networks,num_networks))

    tmpq = []
    tmpq.extend(pvals[ruledim][triu_indices])
    tmpq.extend(pvals[ruledim][tril_indices])
    tmpq = mc.fdrcorrection0(tmpq)[1]
    qmat[ruledim][triu_indices] = tmpq[0:len(triu_indices[0])]
    qmat[ruledim][tril_indices] = tmpq[len(triu_indices[0]):]
    # Find significant values only
    sigflow[ruledim] = qmat[ruledim] < 0.05
    sig_rho[ruledim] = np.multiply(avg_rho[ruledim], sigflow[ruledim])
    tstat_sig[ruledim] = np.multiply(tstats[ruledim], sigflow[ruledim])


I am working on the file "MasterScripts/ManuscriptS1b_Network2NetworkInformationTransfer.ipynb".
The first task is:
2.4 Plot results
Can you write Python code for it?

for ruledim in ruledims:
    fig = plt.figure()
    plt.title('NetworkToNetwork InformationTransfer\n' + ruledim + ' (FDR)', fontsize=18, y=1.04)
    mat = tstat_sig[ruledim]
    np.fill_diagonal(mat,0)
    norm = MidpointNormalize(midpoint=0)
    plt.imshow(mat, origin='lower',norm=norm, vmin=0, cmap='seismic', interpolation='none')
    plt.xticks(netkeys.keys(),netkeys.values())
    plt.yticks(netkeys.keys(),netkeys.values())
    plt.ylabel('Source Network',fontsize=16)
    plt.xlabel('Target Network',fontsize=16)
    plt.tight_layout()
    plt.colorbar()
#     plt.savefig('SFig1_NetworkToNetworkInformationTransfer_' + ruledim + '.pdf')


I am working on the file "MasterScripts/ManuscriptS1b_Network2NetworkInformationTransfer.ipynb".
The first task is:
3.0 Compute significance using FWE correction
3.1 Compute group statistics
Can you write Python code for it?

fwe_Ts = np.zeros((len(networkmappings),len(networkmappings),len(ruledims)))
fwe_Ps = np.ones((len(networkmappings),len(networkmappings),len(ruledims)))

indices = np.ones((len(networkmappings),len(networkmappings)))
np.fill_diagonal(indices,0)
flatten_ind = np.where(indices==1)

rulecount = 0
for ruledim in ruledims:
    t, p = pt.permutationFWE(diff_rho[ruledim][flatten_ind[0],flatten_ind[1],:], permutations=1000, nproc=15)
    fwe_Ts[flatten_ind[0],flatten_ind[1],rulecount] = t
    fwe_Ps[flatten_ind[0],flatten_ind[1],rulecount] = 1.0 - p

    rulecount += 1

pthresh = .05

# Visualize FWER-corrected T-statistic map
rulecount = 0
for ruledim in ruledims:

    # Thresholded T-stat map
    plt.figure()
    # First visualize unthresholded
    mat = fwe_Ts[:,:,rulecount]
    thresh = fwe_Ps[:,:,rulecount] < pthresh
    mat = np.multiply(mat,thresh)
    ind = np.isnan(mat)
    mat[ind]=0
    pos = mat > 0
    mat = np.multiply(pos,mat)
    norm = MidpointNormalize(midpoint=0)
    plt.imshow(mat,origin='lower',norm=norm,vmin = 0,interpolation='none',cmap='seismic')
    plt.colorbar(fraction=0.046)
    plt.title('NetworkToNetwork InformationTransfer\n' + ruledim + ' (FWE)', fontsize=18, y=1.04)
    plt.xticks(netkeys.keys(),netkeys.values())
    plt.yticks(netkeys.keys(),netkeys.values())
    plt.ylabel('Source Network',fontsize=16)
    plt.xlabel('Target Network',fontsize=16)
    plt.tight_layout()
#     plt.savefig('SFig1_NetworkToNetworkInformationTransfer_' + ruledim + '_FWER.pdf')

    # Print out statistics for manuscript
    sig_ind = thresh
    nonsig_ind = thresh==False
    np.fill_diagonal(nonsig_ind,False) # Make sure diagonal is not included

    print ruledim, 'results'
    print 'Average significant ITE effect =', np.mean(avg_rho[ruledim][sig_ind])
    print 'Average significant T-stat =', np.mean(fwe_Ts[:,:,rulecount][sig_ind])
    print 'Maximum significant Pfwe =', np.max(fwe_Ps[:,:,rulecount][sig_ind])
    print '\n'
    print 'Average non-significant ITE effect =',...


I am working on the file "MasterScripts/ManuscriptS4_Network2NetworkInformationTransferNullModel_withinNet.ipynb".
The first task is:
1.0 Run information transfer mapping procedure
1.1 First load in resting-state FC matrices for subjects
Can you write Python code for it?

fcmat = {}
for subj in subjNums:
    fcmat[subj] = np.loadtxt(datadir + 'resultsMaster/MultRegConnRestFC_GlasserParcels/' + subj + '_multregconn_restfc.csv', delimiter=',')


I am working on the file "MasterScripts/ManuscriptS4_Network2NetworkInformationTransferNullModel_withinNet.ipynb".
The first task is:
1.2.2 Run using multiprocessing (parallel processing) to speed-up computation
Can you write Python code for it?

inputs = []
for subj in subjNums:
    inputs.append((subj,fcmat[subj]))

pool = mp.Pool(processes=16)
results = pool.map_async(informationTransferMappingWrapper,inputs).get()
pool.close()
pool.join()

# Collect results
ruledims = ['logic','sensory','motor']
ite_matrix = {}
for ruledim in ruledims:
    ite_matrix[ruledim] = np.zeros((len(networkmappings),len(networkmappings),len(subjNums)))

scount = 0
for result in results:
    for ruledim in ruledims:
        ite_matrix[ruledim][:,:,scount] = result[ruledim]

    scount += 1


I am working on the file "MasterScripts/ManuscriptS4_Network2NetworkInformationTransferNullModel_withinNet.ipynb".
The first task is:
2.0 Calculate group average (True value)
Keep track of networks to matrix indices
Can you write Python code for it?

# Create dictionary that reflects network ordering for matrix rows and columns
netkeys = {0:'vis',1:'smn',2:'con',3:'dmn',4:'fpn', 5:'aud', 6:'dan'}
num_networks=len(netkeys)

baseline = 0.0
avg_rho = {}
tstats = {}
pvals = {}
for ruledim in ruledims:
    avg_rho[ruledim] = np.zeros((num_networks,num_networks))
    tstats[ruledim] = np.zeros((num_networks,num_networks))
    for net1 in netkeys:

        for net2 in netkeys:
            # Skip if net1 and net2
            if net1==net2:
                continue

            # Store results
            avg_rho[ruledim][net1,net2] = np.mean(ite_matrix[ruledim][net1,net2,:])
            tstats[ruledim][net1,net2] = stats.ttest_1samp(ite_matrix[ruledim][net1,net2,:],0)[0]


I am working on the file "MasterScripts/ManuscriptS4_Network2NetworkInformationTransferNullModel_withinNet.ipynb".
The first task is:
2.2 Plot results
Can you write Python code for it?

for ruledim in ruledims:
    plt.figure(figsize=(12,10))
    # First visualize unthresholded results
    plt.subplot(121)
    plt.title('NetworkToNetwork Information Transfer Mapping\n' + ruledim + ' domain', fontsize=14, y=1.04)
    mat = avg_rho[ruledim]
    np.fill_diagonal(mat,0)
    norm = MidpointNormalize(midpoint=0)
    plt.imshow(mat, origin='lower',norm=norm, vmin=0, cmap='seismic', interpolation='none')
    plt.xticks(netkeys.keys(),netkeys.values())
    plt.yticks(netkeys.keys(),netkeys.values())
    plt.ylabel('Source Network',fontsize=16)
    plt.xlabel('Target Network',fontsize=16)
    plt.colorbar(fraction=.046)

    # Next visualize thresholded results (after multiple comparisons)
    plt.subplot(122)
    plt.title('NetworkToNetwork Information Transfer Mapping\n' + ruledim + ' domain' , fontsize=14, y=1.04)
    mat = avg_rho[ruledim]
    thresh = pfwe[ruledim] < 0.05
    # Threshold using q < 0.05
    mat = np.multiply(mat,thresh)
    np.fill_diagonal(mat,0)
    norm = MidpointNormalize(midpoint=0)
    plt.imshow(mat, origin='lower',norm=norm, vmin=0, cmap='seismic', interpolation='none')
    plt.xticks(netkeys.keys(),netkeys.values())
    plt.yticks(netkeys.keys(),netkeys.values())
    plt.ylabel('Source Network',fontsize=16)
    plt.xlabel('Target Network',fontsize=16)
    plt.colorbar(fraction=.046)
    plt.tight_layout()
    plt.savefig('NetworkToNetworkITE_' + ruledim + '_PermutationTestResults.pdf')


I am working on the file "MasterScripts/ManuscriptS4_Network2NetworkInformationTransferNullModel_withinNet.ipynb".
The first task is:
3.0 Load in null distribution for each network-to-network transfer
This was performed on a supercomputer due to computational expense. Code for these permutations can be found at: ./SupercomputerScripts/SFig4_FCPermutationTest
Can you write Python code for it?

nPermutations = 1100
permdata = '/projects2/ModalityControl2/data/resultsMaster/ManuscriptS4_NetworkToNetworkITE_NullModel/PermutationTests_WithinNetPerm/'
cdf_val = {}
pmat = {}
for ruledim in ruledims:
    cdf_val[ruledim] = np.zeros((num_networks,num_networks))
    pmat[ruledim] = np.zeros((num_networks,num_networks))

    null_rhos = np.zeros((num_networks,num_networks,nPermutations))
    print 'Computing p-stats for', ruledim

    for i in range(nPermutations):
        null_rhos[:,:,i] = np.loadtxt(permdata + 'NetworkITE_NullModel_' + ruledim + '_Permutation' + str(i) + '.csv',
                                    delimiter=',')

    for net1 in netkeys:

        for net2 in netkeys:
            # Skip if net1 and net2
            if net1==net2:
                continue

            # Store results
            # Compute ecdf for each net1 - net2 config
            ecdf = ECDF(null_rhos[net1,net2,:])
            # Now find where in the empirical distribution the true rho value lies
            cdf_val[ruledim][net1,net2] = ecdf(avg_rho[ruledim][net1,net2])
            # Since p-values are inverted... and we're only doing 1-sided tests
            pmat[ruledim][net1,net2] = 1.0 - ecdf(avg_rho[ruledim][net1,net2])


I am working on the file "MasterScripts/ManuscriptS5b_PerformanceDecoding_withITE.ipynb".
The first task is:
0.2 Verify normality of behavioral data
Can you write Python code for it?

acc = []
rt = []
for subj in subjNums:
    acc.append(np.mean(behavdata[subj]['acc']=='Correct'))
    rt.append(np.mean(behavdata[subj]['rt']))

plt.figure()
plt.hist(acc)
plt.title('Distribution of accuracies across 32 subjects\nNormalTest p = '+str(stats.normaltest(acc)[1]),
          y=1.04,fontsize=16)
plt.xlabel('Avg Accuracy Per Subj')
plt.ylabel('# in bin')

plt.figure()
plt.hist(rt,bins=10)
plt.title('Distribution of RTs across 32 subjects\nNormalTest p = '+str(stats.normaltest(rt)[1]),
          y=1.04,fontsize=16)
plt.xlabel('Avg RT per Subj')
plt.ylabel('RT')


I am working on the file "MasterScripts/ManuscriptS5b_PerformanceDecoding_withITE.ipynb".
The first task is:
1.0 Run Information transfer mapping
Due to obvious computational constraints, all region-to-region ActFlow procedures and RSA analyses were run on NM3
Code: ./SupercomputerScripts/Fig6_RegionToRegionInformationTransferMapping/ActFlow_ITE_DecodePerformance_LogRegression_v2.m
2.0 Identify information transfers that are correlated with behavior
Load in logistic regression betas
Can you write Python code for it?

## Load in NM3 Data
datadir = '/projects2/ModalityControl2/data/resultsMaster/Manuscript6andS2and7_RegionToRegionITE/'
# Load in RSA matrices
logitBetas= np.zeros((nParcels,nParcels,len(subjNums)))
scount = 0
for subj in subjNums:
    filename = datadir +subj+'_RegionToRegionActFlowGlasserParcels_BehavioralAcc.csv'
    logitBetas[:,:,scount] = np.loadtxt(filename, delimiter=',')
    scount += 1


I am working on the file "MasterScripts/ManuscriptS5b_PerformanceDecoding_withITE.ipynb".
The first task is:
Do the same for RT
Can you write Python code for it?

n_mbs = 128
ntrialspermb = 3
for subj in subjNums:
    filename = '/projects2/ModalityControl2/data/resultsMaster/behavresults/' + subj+'_rt_by_mb.txt'
    tmp = behavdata[subj]['rt']
    mb_tmp = []
    count = 0
    for mb in range(n_mbs):
        mb_tmp.append(np.mean(tmp[count:(count+ntrialspermb)]))
        count += ntrialspermb
    mb_tmp = np.asarray(mb_tmp)
    np.savetxt(filename, mb_tmp)


I am working on the file "MasterScripts/ManuscriptS5b_PerformanceDecoding_withITE.ipynb".
The first task is:
FWE correction on results
Perform group statistics
Can you write Python code for it?

df_stats = {}
df_stats['logit_t'] = np.zeros((nParcels,nParcels))
df_stats['logit_p'] = np.ones((nParcels,nParcels))
df_stats['logit_q'] = np.ones((nParcels,nParcels))
df_stats['logit_avg'] = np.mean(logitBetas,axis=2)
for i in range(nParcels):
    for j in range(nParcels):
        if i==j: continue
        t, p = stats.ttest_1samp(logitBetas[i,j,:],.5)
        if t > 0:
            p = p/2.0
        else:
            p = 1.0 - p/2.0
        df_stats['logit_t'][i,j] = t
        df_stats['logit_p'][i,j] = p

roi = 260
# indices = np.where((networkdef==networkmappings['fpn']) | (networkdef==networkmappings['con']))[0]
# # indices = np.where(networkdef==networkmappings['con'])[0]
fpn_ind = np.where(networkdef==networkmappings['fpn'])[0]
fpn_ind.shape = (len(fpn_ind),1)
con_ind = np.where(networkdef==networkmappings['con'])[0]
con_ind.shape = (len(con_ind),1)
# indices = np.arange(nParcels)
mat = np.zeros((nParcels,nParcels))

# mat[260,:] = 1
# mat[:,260] = 1
mat[260,fpn_ind.T] = 1
mat[260,con_ind.T] = 1
# mat[fpn_ind,con_ind.T] = 1
# mat[fpn_ind,fpn_ind.T] = 1
# mat[con_ind,con_ind.T] = 1
# mat[con_ind,fpn_ind.T] = 1
# mat = np.ones((nParcels,nParcels))
# mat[:,roi] = 1
np.fill_diagonal(mat,0)
indices = np.where(mat==1)

# Permutation testing
import permutationTesting as pt
chance = .5
tmp = logitBetas - chance # Subtract decoding accuracies by chance
t, p = pt.permutationFWE(tmp[indices[0],indices[1],:], permutations=1000, nproc=15)
pfwe = np.zeros((nParcels,nParcels))
tfwe = np.zeros((nParcels,nParcels))
pfwe[indices] = p
tfwe[indices] = t

ind = np.where(pfwe>0.95)
print 'Significant transfers predictive of performance:'
count = 0
for i in range(len(ind[0])):
    print 'Transfers from', ind[0][count], 'to', ind[1][count]
    print 'Effect size =', df_stats['logit_avg'][ind[0][count],ind[1][count]]
    print 'T-statistic =', tfwe[ind[0][count],ind[1][count]]
    print 'p (FWE) =', 1.0 - pfwe[ind[0][count],ind[1][count]]
    count += 1


I am working on the file "MasterScripts/ManuscriptS5b_PerformanceDecoding_withITE.ipynb".
The first task is:
Perform group statistics
Can you write Python code for it?

df_stats = {}
df_stats['logit_t'] = np.zeros((nParcels,nParcels))
df_stats['logit_p'] = np.ones((nParcels,nParcels))
df_stats['logit_q'] = np.ones((nParcels,nParcels))
df_stats['logit_avg'] = np.mean(logitBetas,axis=2)
for i in range(nParcels):
    for j in range(nParcels):
        if i==j: continue
        t, p = stats.ttest_1samp(logitBetas[i,j,:],.5)
        if t > 0:
            p = p/2.0
        else:
            p = 1.0 - p/2.0
        df_stats['logit_t'][i,j] = t
        df_stats['logit_p'][i,j] = p

roi = 260
# indices = np.where((networkdef==networkmappings['fpn']) | (networkdef==networkmappings['con']))[0]
# # indices = np.where(networkdef==networkmappings['con'])[0]
fpn_ind = np.where(networkdef==networkmappings['fpn'])[0]
fpn_ind.shape = (len(fpn_ind),1)
con_ind = np.where(networkdef==networkmappings['con'])[0]
con_ind.shape = (len(con_ind),1)
# indices = np.arange(nParcels)
mat = np.zeros((nParcels,nParcels))

# mat[260,:] = 1
# mat[:,260] = 1
mat[260,fpn_ind.T] = 1
mat[260,con_ind.T] = 1
# mat[fpn_ind,con_ind.T] = 1
# mat[fpn_ind,fpn_ind.T] = 1
# mat[con_ind,con_ind.T] = 1
# mat[con_ind,fpn_ind.T] = 1
# mat = np.ones((nParcels,nParcels))
# mat[:,roi] = 1
np.fill_diagonal(mat,0)
indices = np.where(mat==1)

# Perform multiple comparisons
#     df_stats['logit_q'][sig_ind] = mc.fdrcorrection0(df_stats['logit_p'][sig_ind])[1]
df_stats['logit_q'] = np.ones((nParcels,nParcels))
df_stats['logit_q'][indices] = mc.fdrcorrection0(df_stats['logit_p'][indices])[1]

ind = np.where(df_stats['logit_q']<0.05)
print 'Significant transfers predicted of performance:'
count = 0
for i in range(len(ind[0])):
    print 'Transfers from', ind[0][count], 'to', ind[1][count]
    print 'Effect size =', df_stats['logit_avg'][ind[0][count],ind[1][count]]
    print 'p =', df_stats['logit_p'][ind[0][count],ind[1][count]]
    print 'q =', df_stats['logit_q'][ind[0][count],ind[1][count]]
    count += 1


I am working on the file "MasterScripts/SupercomputerScripts/SFig4_FCPermutationTest/.ipynb_checkpoints/Network2NetworkInformationTransfer_PermutationTest-checkpoint.ipynb".
The first task is:
1.2 Perform network-to-network information transfer mapping procedure using python module
1.2.1 First construct a wrapper to pass through a multiprocessing scheme
Can you write Python code for it?

def informationTransferMappingWrapper((subj,fcmat)):
    """
    A wrapper so we can use multiprocessing to run subjects in parallel
    """
    out = n2n.networkToNetworkInformationTransferMapping(subj,fcmat,null=False)
    return out


I am working on the file "MasterScripts/SupercomputerScripts/SFig4_FCPermutationTest/.ipynb_checkpoints/Network2NetworkInformationTransfer_PermutationTest-checkpoint.ipynb".
The first task is:
1.2.2 Run using multiprocessing (parallel processing) to speed-up computation
Can you write Python code for it?

inputs = []
for subj in subjNums:
    inputs.append((subj,fcmat[subj]))

pool = mp.Pool(processes=32)
results = pool.map_async(informationTransferMappingWrapper,inputs).get()
pool.close()
pool.join()

# Collect results
ruledims = ['logic','sensory','motor']
ite_matrix = {}
for ruledim in ruledims:
    ite_matrix[ruledim] = np.zeros((len(networkmappings),len(networkmappings),len(subjNums)))

scount = 0
for result in results:
    for ruledim in ruledims:
        ite_matrix[ruledim][:,:,scount] = result[ruledim]

    scount += 1


I am working on the file "MasterScripts/SupercomputerScripts/SFig4_FCPermutationTest/.ipynb_checkpoints/Network2NetworkInformationTransfer_PermutationTest-checkpoint.ipynb".
The first task is:
2.0 Compute group statistics
Keep track of networks to matrix indices
Can you write Python code for it?

# Create dictionary that reflects network ordering for matrix rows and columns
netkeys = {0:'vis',1:'smn',2:'con',3:'dmn',4:'fpn', 5:'aud', 6:'dan'}
num_networks=len(netkeys)

baseline = 0.0
avg_rho = {}
tstats = {}
pvals = {}
for ruledim in ruledims:
    avg_rho[ruledim] = np.zeros((num_networks,num_networks))
    tstats[ruledim] = np.zeros((num_networks,num_networks))
    pvals[ruledim] = np.zeros((num_networks,num_networks))

    for net1 in netkeys:

        for net2 in netkeys:
            # Skip if net1 and net2
            if net1==net2:
                avg_rho[ruledim][net1,net2] = np.nan
                tstats[ruledim][net1,net2] = np.nan
                pvals[ruledim][net1,net2] = np.nan
                continue

            # Store results
            avg_rho[ruledim][net1,net2] = np.mean(ite_matrix[ruledim][net1,net2,:])
            t, p = stats.ttest_1samp(ite_matrix[ruledim][net1,net2,:],0)
            # One-sided t-test
            tstats[ruledim][net1,net2] = t
            if t>0:
                p=p/2.0
            else:
                p = 1-p/2.0
            pvals[ruledim][net1,net2] = p


I am working on the file "MasterScripts/SupercomputerScripts/SFig4_FCPermutationTest/.ipynb_checkpoints/Network2NetworkInformationTransfer_PermutationTest-checkpoint.ipynb".
The first task is:
2.1 Perform multiple comparisons (using false discovery rate)
Can you write Python code for it?

# Compute group stats
baseline = 0.0
triu_indices = np.triu_indices(len(networkmappings),k=1)
tril_indices = np.tril_indices(len(networkmappings),k=-1)

qmat = {}
for ruledim in ruledims:

    qmat[ruledim] = np.zeros((num_networks,num_networks))

    tmpq = []
    tmpq.extend(pvals[ruledim][triu_indices])
    tmpq.extend(pvals[ruledim][tril_indices])
    tmpq = mc.fdrcorrection0(tmpq)[1]
    qmat[ruledim][triu_indices] = tmpq[0:len(triu_indices[0])]
    qmat[ruledim][tril_indices] = tmpq[len(triu_indices[0]):]


I am working on the file "MasterScripts/SupercomputerScripts/SFig4_FCPermutationTest/.ipynb_checkpoints/Network2NetworkInformationTransfer_PermutationTest-checkpoint.ipynb".
The first task is:
2.2 Plot results
Can you write Python code for it?

for ruledim in ruledims:
    plt.figure(figsize=(12,10))
    # First visualize unthresholded results
    plt.subplot(121)
    plt.title('NetworkToNetwork Information Transfer Mapping\n' + ruledim + ' domain', fontsize=14, y=1.04)
    mat = avg_rho[ruledim]
    np.fill_diagonal(mat,0)
    norm = MidpointNormalize(midpoint=0)
    plt.imshow(mat, origin='lower',norm=norm, vmin=0, cmap='seismic', interpolation='none')
    plt.xticks(netkeys.keys(),netkeys.values())
    plt.yticks(netkeys.keys(),netkeys.values())
    plt.ylabel('Source Network',fontsize=16)
    plt.xlabel('Target Network',fontsize=16)
    plt.colorbar(fraction=.046)

    # Next visualize thresholded results (after multiple comparisons)
    plt.subplot(122)
    plt.title('NetworkToNetwork Information Transfer Mapping\n' + ruledim + ' domain' , fontsize=14, y=1.04)
    mat = avg_rho[ruledim]
    thresh = qmat[ruledim] < 0.05
    # Threshold using q < 0.05
    mat = np.multiply(mat,thresh)
    np.fill_diagonal(mat,0)
    norm = MidpointNormalize(midpoint=0)
    plt.imshow(mat, origin='lower',norm=norm, vmin=0, cmap='seismic', interpolation='none')
    plt.xticks(netkeys.keys(),netkeys.values())
    plt.yticks(netkeys.keys(),netkeys.values())
    plt.ylabel('Source Network',fontsize=16)
    plt.xlabel('Target Network',fontsize=16)
    plt.colorbar(fraction=.046)
    plt.tight_layout()


I am working on the file "MasteringML_wSkLearn/01_Linear_Regression.ipynb".
The first task is:
Calculating variance of X and co-variance of X and y
Can you write Python code for it?

xm = np.mean(X)
print(xm)

variance = (np.sum((X - xm)**2))/4
print(variance)

# numpy func np.var
print(np.var(X, ddof=1))  #ddof - bessels corelation

ym = np.mean(y)
print(ym)

covar = np.sum((X-xm)*(y-ym))/4
print(covar)

# numpy func np.cov
print(np.cov([6,8,10,14,18], [7,9,13,17.5,18])[0][1])


I am working on the file "MasteringML_wSkLearn/01_Linear_Regression.ipynb".
The first task is:
Performance measures, bias, and variance
There are two fundamental causes of prediction error: a model's bias and its variance.

Bias A model with a high bias will produce similar errors for an input regardless of the training set it was trained with; the model biases its own assumptions about the real relationship over the relationship demonstrated in the training data.
variance    A model with high variance, conversely, will produce different errors for an input depending on the training set that it was trained with. A model with high bias is inflexible, but a model with high variance may be so flexible that it models the noise in the training set. That is, a model with high variance over-fits the training data, while a model with high bias under-fits the training data.
It can be helpful to visualize bias and variance as darts thrown at a dartboard. Each dart is analogous to a prediction from a different dataset.

high bias but low variance -   A model with high bias but low variance will throw darts that are far from the bull's eye, but tightly clustered.
 high bias and high variance -    A model with high bias and high variance will throw darts all over the board; the darts are far from the bull's eye and each other.
low bias and high variance -    A model with low bias and high variance will throw darts that are closer to the bull's eye, but poorly clustered. Finally, a model with low bias and low variance will throw darts that are tightly clustered around the bull's eye,
Ideally, a model will have both low bias and variance, but efforts to decrease one will frequently increase the other. This is known as the bias-variance trade-off.
accuracy, precision, and recall
$$ ACC = \frac{TP + TN}{TP + TN + FP + FN} $$
$$ P = \frac{TP}{TP + FP} $$
$$ R = \frac{TP}{TP + FN} $$
Can you write Python code for it?

# RSS
from sklearn import metrics

print("Mean Abs Error", metrics.mean_absolute_error(y_test, y_predict))
print("Sqred Abs Error", metrics.mean_squared_error(y_test, y_predict))
print("", lReg.score(X_test, y_test))


I am working on the file "MasteringML_wSkLearn/01_Linear_Regression.ipynb".
The first task is:
Plotting
Can you write Python code for it?

# creating figure
plt.figure(figsize=(6, 4), dpi=120)
#plt.plot(x, Y, color='blue')
plt.scatter(X, y, color='blue')
plt.xlabel("Pizza diameter")
plt.ylabel("Pizza price $")
plt.title("Pizza price analysis")
#plt.xlim(0, 30)
#plt.ylim(0, 30)
plt.grid(True, color='0.2')
plt.autoscale(True)


I am working on the file "MasteringML_wSkLearn/01_Linear_Regression.ipynb".
The first task is:
now, calculating coeff - $$ \frac{cov(X,y)}{var(X)} $$
Can you write Python code for it?

coeff = covar / variance
coeff

# based on coeff we can calc intercept which is y - coeff*x

intercept = ym - coeff*xm
intercept

print(coeff, intercept)
print(lReg.coef_, lReg.intercept_)

# checking out the 16" pizza  price
price = 1.96551724138 + (0.976293103448 * 16)
print(price)
print(lReg.predict([[16]]))

# let's test this model on test data
X_test = [[8],[9],[11],[16], [12]]
y_test = [[11],[8.5],[15], [18],[11]]

y_predict = lReg.predict(X_test)
y_predict


I am working on the file "MasteringML_wSkLearn/02a_Classification.ipynb".
The first task is:
Evaluating the model
R-squared measures how well
the observed values of the response variables are predicted by the model. More
concretely, r-squared is the proportion of the variance in the response variable that
is explained by the model. An r-squared score of one indicates that the response
variable can be predicted without any error using the model. An r-squared score of
one half indicates that half of the variance in the response variable can be predicted
using the model. There are several methods to calculate r-squared. In the case of
simple linear regression, r-squared is equal to the square of the Pearson product
moment correlation coefficient, or Pearson's r.
$$ R^2 = 1 - \frac{SS_{res}}{SS_{tot}} $$
where
$$ SS_{res} = \sum_{i=1}^{n} (y_i - f(x_i))^2  $$
$$ SS_{tot} = \sum_{i=1}^{n} (y_i - \overline{y})^2  $$
Using this method, r-squared must be a positive number between zero and one.
This method is intuitive; if r-squared describes the proportion of variance in the
response variable explained by the model, it cannot be greater than one or less than
zero. Other methods, including the method used by scikit-learn, do not calculate
r-squared as the square of Pearson's r, and can return a negative r-squared if the
model performs extremely poorly.
Can you write Python code for it?

SSres = np.mean(np.square(y-lReg.predict(X)))
print(SSres)
SStot = np.mean(np.square(y-np.mean(y)))
print(SStot)

Rsqr = 1 - (SSres/SStot)
print("R sqare score:", Rsqr)
print("R sqare score:",lReg.score(X, y))


I am working on the file "MasteringML_wSkLearn/02a_Classification.ipynb".
The first task is:
Let's try with 9 degree polinomial
Can you write Python code for it?

lReg9 = LinearRegression()

# feature processing
nine = PolynomialFeatures(degree=9)
X_nine_train = nine.fit_transform(X)
X_nine_test = nine.fit_transform(X_test)

#fitting the model
lReg9.fit(X_nine_train, y)

xx_nine = nine.fit_transform(xx.reshape(xx.shape[0],1))
yy = lReg9.predict(xx_nine)
#print(xx_nine)

print("R squared for 9 degree polynomial fit : ", lReg9.score(X_nine_test, y_test))

ax.plot(xx, yy, linestyle='-', c='r', )
ax.legend([ 'Linear ', 'Poly 2 degree', 'Poly 9 degree'], loc='upper center')

fig

# R square value is negative which indicates the poor performance